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Dedication 


This volume is dedicated to Ray A. Kunze in recognition of his long and dis- 
tinguished career as a leading researcher in group representations and harmonic 
analysis. 

Ray Alden Kunze was born March 7, 1928, in Des Moines, Iowa, but lived 
much of his youth in the area around Milwaukee, Wisconsin. He attended Deni- 
son University for two years before transferring to the University of Chicago, 
where he received his Bachelor of Science and Master of Science degrees, both in 
mathematics. Ray’s graduate studies were interrupted during the early 1950’s 
by military service, during which period he served as a mathematical analyst in 
the Department of Defense. Upon completion of his tour of duty, Ray returned 
to his doctoral studies at the University of Chicago, and in 1957 received his 
Doctor of Philosophy degree in mathematics. His dissertation was written under 
the supervision of Irving E. Segal. 

Ray then went to the Massachusetts Institute of Technology as a C.L.E. Moore 
Instructor, and over the ensuing years has served on the faculties of Brandeis 
University, Washington University, the University of California at Irvine, and the 
University of Georgia. At both Irvine and Georgia he chaired the Department 
of Mathematics. Ray has also held numerous visiting positions in the United 
States and abroad. 

Ray’s mathematical research is characterized by exquisite mathematical taste, 
deep creative insight, algebraic elegance, and analytical power. His numerous 
publications include seminal contributions to almost every aspect of group rep- 
resentations and noncommutative harmonic analysis: Fourier transforms on lo- 
cally compact groups, noncommutative interpolation theory, intertwining oper- 
ators for the principal series and uniformly bounded representations of semi- 
simple Lie groups, operator-valued reproducing kernels, holomorphic multiplier 
representations, generalized Bessel functions and Hankel transforms, analysis on 
matrix space, the holomorphic discrete series and related representations, the 
metaplectic (or Segal-Shale-Weil, or oscillator) representation, algebraic induc- 
tion, invariant theory, and exceptional representations. 

Ray is also well-known for his clarity of mathematical exposition. His cele- 
brated textbook on linear algebra, coauthored with Kenneth M. Hoffman, has 
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been a classic for over three decades and has been translated into several lan- 
guages. As further testimonial to his reputation as a lecturer, Ray frequently 
has been invited to teach graduate level summer school courses, for example at 
NSF and NATO institutes and at universities in Italy, France, and Australia. 

No summary of Ray’s professional contributions would be complete without 
taking special note of his extraordinary talents as a mentor of doctoral students. 
Those who have had the pleasure and privilege of being his student have had 
the best mathematical training possible. His immense patience in guiding a 
esudent from course work to independent research, his clear and precise expla- 
nations, his unerring ability to direct a student toward mainstream problems of 
importance on which one could make steady progress, his unwavering support 
and encouragement when the going gets rough, his network of contacts through 
which he helps a student find a position upon completion of the degree, and the 
lifelong mutual affection and friendship that characterizes his relationship with 
his students, make Ray a perfect model of what a thesis advisor can and should 
be. 

With pride and great pleasure, we dedicate this volume to Ray as a tribute to 
his distinguished career as a research mathematician and inspirational teacher. 
We wish him many more years of good health, happiness, and continued deep 
and important contributions to mathematics. 


Preface 


This volume is an outgrowth of the special session on representation theory 
and harmonic analysis held in honor of Ray Kunze at the 889th meeting of the 
American Mathematical Society on January 12-15, 1994, at Cincinnati, Ohio. 
The mathematicians who gathered together for the special session included many 
of Ray’s former Ph.D. students, research collaborators, and others whose work 
has been influenced either directly or indirectly through their associations with 
Ray. As the session progressed, it became clear that the papers being presented 
spanned a wide spectrum of group representations and applications that would 
be of interest to a more extensive audience. 

For the most part, the papers that appear here are based upon lectures pre- 
sented at the special session.! Although the central theme is group representa- 
tions and its applications to harmonic analysis, this volume contains a diversity 
of mathematics. A glance through the table of contents, for example, reveals 
that the algebraic and geometric points of view in representation theory are also 
greatly in evidence, and that quite classical analysis in higher dimensions, not 
particularly closely tied to representation theoretic ideas, appears as well. To 
convey both the richness and extent of the mathematics included in the volume, 
we note that the papers relate to many topics, such as the following: analysis, al- 
gebra, and geometry on Riemannian, pseudo-Riemannian, and locally Riemann- 
ian symmetric spaces; Lie algebra cohomology; projective and integral geometry; 
representations of semi-simple Lie groups, including the holomorphic discrete se- 
ries and highest weight representations more generally, non-holomorphic discrete 
series, the principal series, and exceptional representations; invariant theory and 
the oscillator representation; invariant differential operators and the work of 
Harish-Chandra; p-adic representation theory; and classical Fourier analysis. 

In closing, the editors are pleased to recognize and thank two other individuals 
who helped produce this volume: Barbara Palmore, whose excellent typing and 


1A few of the lectures treated research that had already been submitted elsewhere, and of 
course could not be included in this volume. However, in some cases the topic does appear in 
the volume as an expository overview of the subject that is not available elsewhere. Conversely, 
some invited speakers who were unable to attend the special session submitted their work in 
written form to these proceedings. 
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formatting of the manuscripts is reflected in the finished product, and Donna 
Harmon of the A.M.S. staff who provided expert production advice. We also 
wish to thank the referees for their careful reading of the papers and their helpful 
comments to authors. Most importantly, the editors are indebted to the speakers 
at the special session for helping make the event so memorable, and for their 
encouragement to produce this volume. 


Tuong Ton-That 
Kenneth I. Gross 
Donald St. P. Richards 
Paul J. Sally, Jr. 
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Supercuspidal character formulas for GL, 


Lawrence Corwin', Allen Moy, and Paul J. Sally, Jr. 


0. Introduction 


The irreducible unitary representations of the multiplicative group D* of a 
division algebra of degree n (dimension n”) over a p-adic field F were determined 
by Corwin [1] and Howe [7] in the case when (n,p) = 1. Corwin and Howe showed 
that the representations are parametrized by admissible characters of extensions 
E/F of degree m where m|n. In a subsequent paper [2], Corwin and Howe began a 
study of the characters of these representations. Any extension E'/F of the above 
type can be embedded in both D* and GL, (F), and the matching theorem states 
that the irreducible characters of DX on E™ are equal, up to a sign, to the discrete 
series characters of GL,(F) on E* regarded as an elliptic Cartan subgroup in 
GL, (F). Thus, the discrete series characters for GL, (F’) on the elliptic set can be 
obtained from the irreducible characters of D*. The full details will be presented 
in a paper by the authors [3]. 

There is little quantitative information available on the discrete series charac- 
ters for reductive p-adic groups. The characters for SL2(F’) [12], PGL2(F) [14], 
and GL2(F) [13] have been known for some time in the case p # 2. Kutzko [9] 
determined the supercuspidal characters of GLy(F’), £ prime, without any restric- 
tion on the characteristic of F. Kutzko’s results take a form which is different from 
those in the present paper since the characters are not readily expressible as locally 
constant functions on tori when p = @. We also note the character formulas in 
Gérardin [5] which hold in a limited range for unramified supercuspidal represen- 
tations of GL,(F’) when (n,p) = 1. The reader should consult the paper [4] of the 
authors for more background and details, and the paper [11] for some additional 
comments. A different perspective on characters, following the work of Bernstein, 
Kazhdan, and Lusztig may be found in the paper [6] of Hales. 

In this note, we illustrate the procedure used in the computation of supercus- 
pidal characters for GL,. In particular, we consider the case in which n = @, a 
prime, for which the complications are relatively minor. Assume that D is a divi- 
sion algebra of index ¢ over F’. Let F be the residue class field of F', D the residue 
class field of D, O the integers in D, and P the prime ideal in O. Let [EF : F] = 4, 
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with FE totally ramified over F. We start with an admissible character w on E™* 
and construct 7 = 71, € (D*)~. We shall then compute the character 9, of 7. 
We embed FE in D as follows: there is a prime w € D with wé € F, and there 
is a set D~ of representatives for D = O/P such that the following statements are 
true: 
(a) 0 € D~ and D~ \ {0} is a cyclic group; 
(b) The map at—> waw7! is bijective on D~; 
(c) If @ +> @ is the restriction of the map O —> O/P =D to D~, then @ +> 
waw! generates Gal(D/F); 
(d) Flaw] & E. 
(See, e.g., [15].) We define o € Gal(D/F) by @ = (waw!). In fact, o determines 
the algebra D. 
We shall often regard F' as F'|w] and write a for @, as a matter of typographical 
convenience. However, we shall be careful not to confuse a and @ in calculations. 
The authors would like to express their thanks to Jeff Adler for his helpful and 
incisive comments. 





1. Construction of the representations and the inducing characters 


Let v be the usual valuation on D (with v(w) = 1). Recall that every element 
xz € D has a unique expansion in the form 


[o.@) 
r= > a;w, a; € D~ and jo = v(x) (hence a;, # 0). 
J=Jo 
Let FY = F 1 D~. Then it is easy to see that 
re k<=>a;€F forall j. 


Let g = card F~ = cardF. 

Note that Trp/r: D — F is faithful on F (in fact, it’s just multiplication by 
é there.) Let T = {y¥ € D| Trp/rY = O}, and let T~ be the preimage of T in D~. 
Then T@F=D. 

We now construct 7. Recall from [8] that (in our present, prime-dimensional, 
totally ramified case) a character y of E* is admissible over F if and only if wli+p, 
does not factor through the norm from F& to F. Any admissible character has a 
Howe factorization ; 

w= Vi(y2oNzye), 
where wy is chosen to minimize the conductor of y, (see, for example, [10]). For 
simplicity, we shall assume that y already has minimal conductor, so that ~ = 1 
and twisting by w2 is unnecessary. (We observe that twisting by #2 only changes 
the central character of the resulting representation of D*.) This means that the 
conductor m+ 1 of y satisfies 
(m, @) = 1. 
Now consider ¥|(14p-)nge- By the definition of m, Wiaipmt1jnz = 1. Therefore, 
Yl14pm is determined by ¥(1t+aw™), a € FY. Thus there is an additive character 
wp of F such that 
pl +aw™) = ¥o(Tra) = Yo(la); 
Moreover, wo is nontrivial. 


SUPERCUSPIDAL CHARACTER FORMULAS FOR GL, 3 


Now define 
m’ =m" = mt if m is odd; 
m'=F+1, m”= 4 if m is even; 
let H’ = EX(1+ P™), H” = EX(1+ P™"), and H = E*(1+ P). We want to 
extend 7 to w! on H’. For this purpose, we set 


Ko = {ye P™ | Trojp yz = 0 for all z € E}, 


and define 
K = (1+ P™*!).(1+Ko). 


Then K is a normal subgroup of H’, and KN EX C1+P™*!. We observe that 
H'=K. E™, and we extend w by setting 


wl = 1, w" | ax = y. 


Note that p#(1+ Bw™) = vo o Trp/r(f) for all 6 € DY, since D=F OT. It will 
be useful later to know that if y € Ko, with y= °°, yj; (here, jo > m’), then 
Yio E r™. 

Next, we extend w! to ~~ on H”. When m is odd, this is trivial, since H’ = H”. 
When m is even, the extension can be done via a Weil representation (as in [10]). 
In this case, we need the following facts about ~~: 

1. dim = q‘@ (recall g = cardF; q’-! = card T). 

2. On FX - (14+ P) NA’, y~ is a multiple of y*. 

3. On elements of E' of the form z = agw??(l+a,w+-:-), with agw” ¢ F, 

we have (when q is odd) 


v (2) = NE/F)v(2), 


where, for E/F ramified, \(E/F) = 1 if q is a square mod @ and —1 other- 
wise. (See [10]}.) 


Define p = Indjj, ¥~, 7 = Ind p. We want to compute ©O,. There are three 
main stages in this procedure: 
(i) computing 6,~ from 7%, 
(ii) computing @, from 6,~, and 
(iii) computing ©, from 4). 
The first stage (or, at least, as much as we shall need of 6,,~) amounts to the three 
facts about ~~ listed above. 
The second stage is the longest and hardest. We break it into steps. 


2. Restricting the support of 6, 


We state a lemma which will be proved in full generality in [3]. Here we will 
be content to prove a special case, which gives the flavor of the general proof. 

We say that z = 57°. ajo (a; € D~ for all j) is normal if the terms ajo! 
all commute. 


LEMMA 2.1. Let D be a division algebra of index n over F, with (n,p) = 1. 
Given y € D, there is a unique normal x € D conjugate to y under 1+ P. 
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PROOF IN A SPECIAL CASE: Assume that y = )°°°, a;07, with a; = 1. We 
show that y is conjugate under 1+ P to a unique x € D with 


c= 5 pw’, all 8, €F~. 
g=l 


The proof is inductive. Conjugate y with an arbitrary element of 1+ P, and com- 
pute mod P?, remembering that wy = ym for y € D~: 


(l+é6m+6'w?+---jy(1t+é6m+6'm?+---)7} 
=(l+ém+---)y(1~(6m+---)+(6m+---)*) 

y+ (6a% — a,67)w? (mod P?) 

=wt(a,+6—67)w? (mod P*), 


since a; = 1. Now in D, 6 — 6” can be any element of T; hence ag + 6 — 6° can be 
any element of D with trace equal to that of aj. Hence there is a unique choice of 
Bo = a2 +6 —6° that is in F (though 6 is not unique). With an appropriate choice 
of 6, then, y is conjugate to yo = w+ fow* +a,m?4+---, with B € F (and py 
uniquely determined). Now repeat, conjugating yo by (1+ 6’w*+---) to produce 


y= wt hw + por +aya*t+---, 


with 2 and #3 in F and uniquely determined. Finally, take the limit of the y; as 
jro. Oo 


It follows that we can restrict attention to normal elements x. Write x in the 
form x = agw?(1+a;m+ aw? +---). When we write a;a/, it is understood 
that this means ayw?? when j = 0. (Note that this is at variance with the previous 
use of indices.) 

We now reimpose the restriction that n = @, a prime. 


LEMMA 2.2. Suppose x is a normal element and x ¢ F. If j is any index such 
that a;w! ¢ F, then F(x] = Fla;w’]. 


Proor. [F[a;] : F] = @, since all extensions of F in D have degree ¢ (except 
F itself). Hence F[a;7’] is its own centralizer in D. It follows that every a,’ is in 
Flaj;w!], and hence x € Fla;w/]. Since [F[z] : F] = ¢, the conclusion follows. O 


LEMMA 2.3. pliqpm is a multiple of w. 


PROOF. Since ~~|14p™ is a multiple of ~, it suffices to show that for some set 
S' of coset representatives for H/H”, 


(2.4) we S => p(wyw") = v(y), allye€1l+P™. 


Since H/H” = (HN(1+P))/(A"”N(14 P)), we can pick the w to bein 1+P. But 
1+ P™*+! is normal in 1+ P, and is contained in the kernel of y. So (1+ P™)/(1+ 
P™*"1) is central in (1+ P)/(1+ P™*?), and (2.4) is obvious. O 


PROPOSITION 2.5. If x is normal and 6,(x) £0, thenz € E*(1+ P™). 


Proor. Note first that the hypothesis and conclusion are unaffected if x is 
multiplied by an element of F*. 

Write x = agw?*(1+a,w+---), and assume that the proposition is false for 
x. Since ajw° € E by construction of H, we must have ajpw° € F; otherwise, 
Lemma 2.2 gives x € E. By the note above, we may divide by apw?*; that is, we 
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may assume that jo = 0 and ay = 1. Let j be the first index with a;w’ ¢ F. Then 
j < m (otherwise z € F(1+ P™)), anda, ¢ FY. 
Now let zs = (1+ 6m™7!)z(1+ 6m™~1)—!, 6 € DY. We have 


(2.6) XE =f ae (6a7"” _ a6” )o™ (mod Pret): 


(2.7) fp =a2(1t+ (6a7"” - aj6” )o™) (mod 1+ P™*?). 


This sort of computation will be needed often in the future; here are the details in 
this case. Note first that 


(1+ 60™-9)-1 = 1-6m™ 5 + (60™ 4)? —--- 


Now compute (1 + 6m@™~J)a;w'(1+ 6@™~)~? for each i. When i < j, a;w? is 
central, and the result is a;w*. When i > j, the expanded product has every term 
except a;a’ in P™*!; that is, 


(1+ 60" J\a,a'(1+ 60")! =ayw* (mod P™*1) fori>j. 
For i = j, the only terms in the expansion that are not in P™+! are 

aw + 6a" Ja, —a;w'dw™ =a;0' + (602" ” - 0,67 \w™, 
because wy = y’m if y € DY. Now summing gives (2.6), and (2.7) follows 


from (2.6) by multiplying out terms. 
Since 6, is constant on conjugacy classes, Lemma 2.3 gives 


6,(x) = 9(25) = 0,(x)b(1 + (6a7"” — 0,67’ )w™), 
for all 6 € D~. By our assumption, 6,(z) # 0. Therefore, we have (for all 6 € D~) 
1= (1+ (607" ” — a;67’)w™) 
= Yo oTrpr(6a7”  — 0367’) 
= Yo 0 Trpjr 6” (a?” — a5), 


this last holding because Trp/p(y) = Trp/r(y7) for all y ¢ D. But yo is nontrivial 
and Trp/p is surjective; thus we must have 

a a =0, a” 
But o” generates Gal(D/F), since @ { m and @ is prime. Therefore G is Gal(D/IF)- 
invariant, or @; € F, or a; € FY. This gives the desired contradiction, and the 
theorem is proved. O 


= Aj. 


3. Computing 6,(z) 


We need to examine @,(x) more closely for zc € E*(1+ P™). In view of 
Lemma 2.2, we can treat E* and 1+ P™ separately, and we have described 6,|14.pm 
in Lemma 2.3. Thus we need only deal with @,| px. 

Given any normal z € D, write z = agw?°(1+a,w+---) and define 


BGS 0 if agw ¢ F; 
aaa j if agw% € F and j is the smallest index with ajw! ¢ F. 
(If z € F, vo(z) = 0.) 


When vo(z) > m, then z € F*(1+ P™) and we have already described 6,(z) 
(modulo knowing dim p). For other z € E*, we need to know more about H/H”. 
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LEMMA 3.1. For y = (11,--- ;¥m—1) € (T~)™'—}, let 


Yy =(l4+y1@)----: (l+%m-10@™ —"). 


The y, form a set of coset representatives for H/H". 


Proor. Let Hj = HN(1+ P?), HY = H"” (1+ P*). Then H; = H? for 
j =m’. We noted in the proof of Lemma 2.3 that H/H” = H,/H/{/. We thus get a 
complete set of coset representatives for H/H” by getting coset representatives for 
H;/ Aj. ¥,1<j<m"—1, and multiplying them together. Since the elements 
1++4;0/, 7; € T~, give such a set of coset representatives, the lemma follows. O 

CoROLLarRY. dim p = q’&-)("-) dim y. 

PRoor. card T~ = g®7!. oO 


PROPOSITION 3.2. If vo(x) =0 (and x € E), then 6,(x) = 0y4~(z). 
PROOF. The standard Frobenius formula gives 
Op(t)= Y. by~(yyzy;"), 
yeTwm’-1 


where by~ = 6,~ on H” and 0 elsewhere. It suffices to prove that ify # (0,... ,0) = 
0, then y,cy>' ¢ H". 

If we write x = agw(l+ajw+-:-), then ¢/ jo. If y £0, let 7, be the first 
nonzero coordinate in y. Then we get 


Uy2Yy" =@ot+ (ys08 - ag?” )aiots (mod Piotstly, 


Let 6 = y,0%° — ag”. If 6 ¢ F, then y,zy,) = 2(1+ az °° $7) (mod 1 + 
Ps+1) ¢ H” (recall that s <_m”), and we are done. 
Now suppose that 6 € F. Since x € E, we have ao € F~; thus 
6= a0(Ys — ye); Trpsr é = 0. 


That is, 6 € T~. Thus, 6 = 0, so y, = y2’°. But, since ¢ J jo, 02° generates 
Gal(D/F); hence y, € F. As y, € T~, we have y, = 0. This contradicts the choice 
of +y;, as desired. O 


PROPOSITION 3.3. Let x € E be such that m > vo(x) = j > 0, and let i be the 
smallest integer with 21+ 7 >m. Set 


eye ys {7 = (5-0 %m—1) € (TY) “| 4, =0 fors< ae 


Then 6,(z) = Do ye(T~)™=1(4) by~ (ines. )- 


REMARK. For 7 € (T~)™’~!(i), since y, = 1 (mod P*~1), we have y,cy;} = 
z (mod P/+*~). But j7 +4 >m’, so that y,ry>! € H”. 


PROOF. Let y be a nonzero element of es tee and let s be the smallest 
index such that 7, # 0. If 7+s < _m”, then an argument like that in Proposition 3.2 
gives y,cy,' ¢ H”. So we may assume that j + s >m". 
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Suppose that m” < j+s < m-—s. (The latter inequality holds whenever s < i.) 
Let k = m-—j-—s >, and consider 


(3.4) S- Oy~ (2692; 1251), z=1+60%. 
seD 


A bit of thought shows that the proposition holds if this sum is always 0 when 
s <1. So we examine (3.4). Dividing by an element of F'* if necessary, we may 
assume that 


c=l+ajo!+---, eyo. 
Then 
= Yy LY, | =a2t+em!*® (mod PIts*!), 
s 3 J 
E= 505 —a;7f =a;(7,—7¢ )- 
Hence 


zsv'zs) =a’ + (6m*s — 6a") + (5e7" - 66°" a (mod P™+?), 
That is, 
xx’ zz) = (2' + bw*x — wb*)(1+ (be7 — 66°" )m™) (mod 1+ P™*?), 
We now examine x’ + 6w*x — xéa*. First of all, 
14+ bdm*x — x6m* € K. 


Why? Note first of all that lowest order term in 6w*x — r6m* = 6m*(x — 1) — 
(x — 1)$m* is dw*ajm!? — ajwidw* = a;(5 — 6°’ )wit*, because a; € FY, and 
jtk>m’. 

Note secondly that if u € EF, then 


Trp/r(6o*r — t60")u = Trp/r(ém*uz — urdm*) = 0 


because Trp; (ab) = Trp; r(ba) and xu = uz. 
Thirdly, note that z = 2’ (mod P+’). Thus, setting w = éw*x — zém*, we 
have 


ge t+w=2'(lt+w)-(2-l)hw=2'(l+w)-(c£-1)w (mod P™*?). 
But c—1¢€ POE, and so it is easy to check that 1+ (x -1)w € K. Thus 
ze +we2'(1t+w)(1—-(z-1)w)-(c£~1)?w (mod P™*), 
and it follows by induction that for some w’ € K, 
(3.5) zg t+we=a2'w' (mod1+P™*), 


It follows that 6,~(z’ + w) = 6,~(z’). (Note that w depends on 6.) Thus we 
conclude that 


By~ (zgx" 251) = Oy~ (2’)(Wo o Trp p(5e” — €67°"”)). 


The sum in (3.4) is therefore a multiple of S?s-p Yoo Trp jp (67 — e6"""), Hence 
the sum is 0 unless for all 6, 


l= wo o Trpr(de”" = 6?""") = Wo oTrp/r(e” = é). 
As usual, this implies that ¢°” =e; and hence that ¢ € F. That is, 


a5 (Ys _ 42") eF. 
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Since a; € F, we also have a;(7, — 2’) €T. Thus a;(y, - 42’) = 0, or 7, = ye". 
As o/ generates Gal(D/F), this means that y, € F. By construction, y, € T. So 
‘Ys = 0, a contradiction; the proposition is proved. O 


PROPOSITION 3.6. For x € E with 0 < wo(z) = j < m, and with i as in 
Proposition 3.3, 
(a) If 2+j>m, then 6,(x) = q@Vir"-49,,. (x); 
(b) If 2i+j7 =m, then 


Pa m"'-i- itz 
Op(z) = gM Y6y~(2) S> doo Trappasy” (77 
yeET~ 


3 


=). 


PROOF. We have 


yyryy,) = 2+ SY (-1)*[(e- (yy -— 1D) - (yy - Y(@-V](y-1)*? 
d=1 


ie 
=x2+ S- Wd say. 
d=1 


Since y;! € P! and x—1€ P?, the we with @ > 2 are all in P™+?. 
(a) If 2i+j > m, then w2 € P+! as well. It is easy to check that Trp/p(wiz) =0 


for all z € E (the argument is the same as one in Proposition 3.3), and w; € P™ 
as well. Thus Dope w,; € Ko, and an argument like that leading to (3.5) gives 


YyLY; | = rk, KEK. 
Hence 6y~ (y,ry7') = 9y~ (x). Now (a) follows from the fact that 
card T™’—1(4) = qi" DED, 


If 21 + 7 =m, then a calculation gives 


fez 
— 


Ww. - ayy (V7 = yi)o”™ € prt. 
(Only the lowest order terms in the expression for w2 are not in P™+!, so that 
the calculation is not too hard.) But this means that (mod 1+ P™*?), 

= t+7 

Yatyy) = (a+ wi) t+ ayy? "(1 — 1%)@")- 

As with (a), 7+ wi = ck, k © K. It is now easy to compute 6,~(y,ry,'); 
summing yields (b) (with +, for y). O 
We will have an estimate for |6,(x)| once we evaluate the sum in (b). 


PROPOSITION 3.7. If vo(z) < m and x is (conjugate to) an element of E*, 
then 


1 
Op(x) = 92 G(x, p)y(z), 
where G(x,w) is a fourth root of unity. Indeed, 
ME/F) if vox) =0; 
1 if 7 > 0, and m and j have opposite parities; 


toe nee 
q eV? S ep boo Mayr (ay “(7 —7)) 
if 7 > 0, and m and j have the same parity. 


(3.8) G(z,¥) = 
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PROOF. Without loss of generality, we may assume x normal. We first consider 
the case when vo(x) = 0. Here, by Proposition 3.2 and Fact 3 (near the end of §1) 
we have 0,(x) = 0y~ (x) = A(E/F)y(z). 
Next, we consider vg (x) > 0. 
(i) If m is odd and vo(x) = 7 is even, then Proposition 3.6(a) applies. We have 
Oy~(x) = pR(x), m” = ™H, and i = =4=2. Hence the exponent of q in 
Proposition 3.6(a) is 


(€—1)(m" — 4) = (1H - 4) = 5-5, 


as desired. Here, G(x, wy) = 1. 

Suppose m is even and vo(x) = j isodd. The hypothesis that m is even implies 
by Fact 3 (near the end of §1) that m” = ™ and 6,~(z) = q'—))/?y(a). The 
j odd hypothesis means i = mio and therefore we are again in the situation 
of Proposition 3.6(a). Hence 


O,(z) =Oy(z)q* = (G(z,¥) = 1), 


(ii 


aes 


where 
a= Sh4(€-1(m" 1) = G+ (€- (8 - BY) = GY 
again. 


If m and j are both odd, we use Proposition 3.6(b) and Proposition 3.9. Since 
Oy~ = w, we see that 


ae 


(iii 


off 
ry 
=] 
jon 
5S 
ll 
i 
| 


where, since m” = 


again. 

If m and j are both even, we need Proposition 3.6(b), Proposition 3.9, and 
the fact that 0,~(a) = q°~))/*y(z). The algebra is routine. 

That G(z,7) is a fourth root of unity is clear in the first two cases of (3.8). The 
proof in the third case of (3.8) is the next proposition. 0 


VS 


(iv 


PROPOSITION 3.9. If q is odd, then 
S> doo Trprlay” (y” — 9) = G2, v)q"Y/?, 
yeT 
where G(x, w) is a fourth root of unity. 
PROOF. Set 
Q(7,) = Troe ly?” (6 — 8) +67 "(9 -y)], 7,8 ET. 
Then the sum we need to evaluate is 


(3.10) S> ¥0(50;Q(7,7))- 

yeT 
By results going back to Gauss, (3.5) is (card T)? G(x, w), where G(z, W) is a fourth 
root of unity, provided that Q is nondegenerate. Rewrite Q as 


Q(7,6) = Tryp 5(y" — 9 4497-97”): 
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Suppose that for some y € T, Q(7,6) = 0 for all 6 € T. Obviously Q(y, 6) = 0 if 
6€F; since D=F OT, we get 


a 


yy ty 97 = 00. 
Using the fact that m = 2i + j, we can rewrite this as 
Gaar)y” SGa4"). 
Since m and ¢ are relatively prime, it follows that y — y?? EF. 


But obviously y — 7%’ € T; thus y — y” = 0. Moreover, since generates 
Gal(D/F), we have y € F. But y € T; hence y = 0. This proves nondegeneracy. O 





4. Computing 0, explicitly 
The following theorem summarizes previous results. 


THEOREM 4.1. Forx € H and q odd, 
(a) If vo(x) > m, then 6,(x) = q2™- DED y# (a). 
(b) If vo(x) <‘m and x is not (conjugate to) an element of E, then 6,(x) =0. 


It is fairly easy to compute 9, from 6,. 


THEOREM 4.2. Let x € D*. 
(a) If vo(x) > m, then ©,(x) = Fa gh VED ap # (x), 
(b) If vo(x) <m and x is not conjugate to an element of E, then 9O,(x) = 0. 
(c) If u(x) <m anda € E, then 
i m— Vol re 
O,(z)=q2i™ 0) Sy, Wd), 
yé(Gal(E/F))-2 


where Gal(E/F) is the group of all automorphisms of E fixing F. (In our case, 
it is {1} unless E is normal, since there are no intermediate fields.) 
(d) If vo(x) =m, thenz =aof, f € F andzp =1+aw™ (mod P™*'). Then 


Lin -1)(e— 
O,(z) = g2"VEVu(f) S> doo Trn/e(8). 
BeD* 
Np/r(B/a)=1 
(Note that a #0, since vo(x) = m.) 


PRoor. The main point to notice is that H is normal in D*, with D*/H = 
D* /F*. The rest is just a calculation with the Frobenius formula 


(4.3) O,(z)= SY) 6,(wrw7'). 
weD*/H 
g 
= 
(Note that ~*(x) = 1; the only point of including it is to make the formula 
valid after w is twisted by a character of F’.) 
(b) This is immediate from (b) of Theorem 4.1. 
(c) The only terms in (4.3) that contribute to the sum nontrivially are those for 
which wrw! € H. In fact, by Theorem 4.1 we can assume wrw™! € EF. Since 
x generates E, if if wrw~! € E, then conjugation by w gives an automorphism 
of E. Furthermore, if conjugation by two elements w and w’ yield the same 
automorphism of E, the fact that E/F is tamely ramified implies w and w’ lie 


(a) If vo(x) > m, then wrw~! € ker p for all w € D*/H, and card D*/H = <=}. 
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in the same H coset in D*. Finally, the Skolem-Noether Theorem says that 
every automorphism of & fixing F' comes from an inner automorphism of D. 
Now (c) is clear. 
Obviously we may restrict to the case f = 1. Choose the w to be in D~™ 
(mod F~*); then (mod 1+ P™*?), if w =7, 


wow! =1+7a(y7")7!o™. 


As y™ generates Gal(D/F), the elements y.(7°" )~! are precisely the elements 3 
of D with Np;r(@/a), each occurring once. That, with (4.3), gives the proof. O 


NoTE. Of course, if x is conjugate to x’ € E, then ©,(r) = 9,(2’). Thus we 


have computed ©, everywhere in D. 
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Geometric Realizations for 
Highest Weight Representations 


M. G. DAVIDSON AND R. C. FABEC 


Dedicated to Ray A. Kunze 


ABSTRACT. Let w be an irreducible unitary representation of a Hermitian 
group G having a highest weight vector. In this paper we construct an 
antiholomorphic mapping from the bounded symmetric domain G/K into 
the space of linear operators from the K linear span H of the highest weight 
vector into the Hilbert space of the unitary representation w. This operator 
is used to obtain the intertwining operator and the unitary structure for 
the geometric realization of w. 


1. Introduction 


The theory of highest weight unitary representations is well-developed alge- 
braically. For Hermitian groups they are described in terms of Verma modules 
and conditions for unitarity are determined by specific algebraic conditions, [5]. 
This article will address how these representations can be naturally realized an- 
alytically on a Hilbert space of holomorphic functions on complex bounded do- 
mains associated with the group. In particular we provide an abstract method 
for obtaining an intertwining operator between any highest weight representa- 
tion and its geometric realization. This is done by determining a natural dense 
subspace of analytic vectors on which the action of the group is given by a com- 
position of an operator valued multiplier and the natural action on the bounded 
symmetric domain. In the literature there are several papers, {1, 2, 9, 10], where 
concrete realizations are considered that fit into this framework. 
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For a unitary highest weight representation w of an Hermitian symmetric 
group G we construct an antiholomorphic mapping T +> qr on the bounded 
domain D = G/K into the set of operators from the irreducible K space H 
consisting of the K span of the highest weight vector for w into the Hilbert 
space H of the representation w. The adjoints gj. of these operators are then 
used to obtain a unitary isomorphism of H into a Hilbert space of holomorphic 
H valued functions on D. Specifically, this mapping is defined by v > E(w) 
where =(v)(T) = g}-v. Furthermore, Q(T, S) = q+-qs defines a positive definite 
operator valued function on D, and this positive definite function when used in 
the sense of Kunze, [8], gives a Hilbert space of holomorphic functions on D on 
which Q(T, S) is the reproducing kernel. This space is precisely the range of =. 
In particular = gives a geometric realization of the representation w on a space 
of holomorphic vector valued functions on D. It is shown this realization can be 
completely described using the operators gr and a multiplier J(g,T) which is 
defined in terms of the irreducible representation of K on H. 

The operator gr is constructed using a power series involving the action of 
the conjugate of T relative to the infinitesimal representation of w. For nonzero 
v in H, it is shown in Theorem 5.1 that the series for grv converges iff T is 
in the domain D. Thus existence of the operator gr corresponds exactly to T 
being located in the domain D = G/K. The operators qr are related to the 
representation w in Theorem 7.1 by the formula 


w(g)arv = 4-7 J(9,T)*~*. 
This is used in showing = intertwines the representation w with the geometric 
representation (7,H(D, H)) where H(D, H) is the Hilbert space determined by 
Q(S,T) and T is the natural geometric representation on this space. 


2. Notation and Preliminaries 


Let G denote a connected noncompact simple Lie group with finite center. 
Let K be a maximal compact subgroup. We assume K has a one-dimensional 
center. This implies that G/K is a Hermitian symmetric space. Let T Cc K bea 
compact Cartan subgroup and let go, &, and tp be the Lie algebras of G, K, and 
T respectively. Take g, = & + po. to be the Cartan decomposition. We adopt 
the convention that the removal of the subscript o will denote complexification. 
Our assumptions imply that there exists a decomposition g = py @&@p_ where 
p, and p_ are abelian subalgebras of p satisfying 


[t, p4] C pa and 
[pP+,p_] Ct. 


Furthermore, if a bar denotes conjugation on g with respect to go, then we 
have 


(2.1) 


(2.2) pi C pz. 
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The subalgebra t is a Cartan subalgebra of g. We let © be the roots corre- 
sponding to the pair (g,t). Let ®. and ®, denote the set of compact and non- 
compact roots, respectively. We denote the root space corresponding to a € ® 
by ga. A positive system of roots 6+ may be chosen so that if 64 = 6+ NS, 
and ®> = (—®+)N®,, then 


P+ = Oy cotBa and 


(2.3) 
p_= Bacer Qa: 


If 6 denotes the Cartan involution on g and B denotes the Killing form, then one 
can choose root vectors Ey, € ga, a € ®, so that B(E,, E_a) = 2/(a,a), and 
6E. = —E_., where (-,-) is the standard inner product on the real space of linear 
functionals on t taking purely imaginary values on t,. We put H, = (Eo, E_4). 
Then we have 


[Ha, Ea] =2E, and 
(0 eee 


Let r be the split rank of G. We inductively define a maximal set {7,..., 
+r} of strongly orthogonal positive noncompact roots by setting 7, to be the 
largest root of ®* and 7,41 the largest root in ®* which is strongly orthogonal 
to y1,°°+,7j- Then Ey, + E_y, lies in po for each 1 <j <r and 


(2.4) a= EG R(E,, + E_,,) 


1<j<r 


is a maximal abelian subspace of po. Let A be the connected subgroup of G whose 
Lie algebra is a,. Then we have the well known decomposition G = KAK. 

We now turn our attention to the Harish-Chandra realization of G/K as a 
bounded symmetric domain in p;. For this it is convenient to assume that G has 
a faithful representation; thus G C Gc. Let P, = exp(pi). Then the mapping 
pi x Kc x p_ — Ge defined by (a,k,b) > exp(a)kexp(b) is a holomorphic 
diffeomorphism onto a dense open set 2 = P, KcP_ of Gc. We uniquely write 
each x € 2 as a product 


(2.5) x= 14(xr)1(x)n_(2), 


where 7i(z) € Pz and m,.(r) € Ke. It is well-known that G C 2 and that 
the map ¢ : 2 — px defined by ¢(z) = log(z4(z)) induces a holomorphic 
diffeomorphism of G/K onto ¢(G). The set D = ¢(G) is a bounded symmetric 
domain in p; which we identify with G/K. 

For each (g,T) € Ge x px for which gexp(T) € 2 we set 


9 T = log m,(gexp(T)) 


(2.6) ; 
i(9,T) = 70(g exp(T)). 
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The map j : Gc x py — Ke is a C™ map and is holomorphic in the p, variable. 
Furthermore, it satisfies the following properties 

i(k, T) = k, 
(2.7) j(p,T) =1, 

j(9192,T) = 5(91, 92 -T)5(92,T), 


where k € Ke, T € pz, p © Pi, 91,92 € G, and for which 9,9. exp(T) and 
g2exp(T) are in 2. 


3. Highest Weight Representations 


Let w be a nontrivial irreducible unitary highest weight representation of G 
on a Hilbert space H with inner product denoted by (-|-) . We will let dw denote 
both the derived representation of w on g, and its complex linear extension to g. 
However, when there is no confusion we will usually write X¢ or X - ¢ instead 
of dw(X)¢, where ¢ is a C™ vector and X is in the universal enveloping algebra 
U(g) of g. Let v. denote the highest weight vector and let H = H, denote the 
K span of vo. Then 4H is an irreducible K-space. Let 7 denote the restriction of 
wl to H and set A to be its highest weight. By (2.1) we also have 


(3.1) T-v=0 


for all T € px and v € H. For each positive integer n we define H, to be the 
subspace of H spanned by all terms of the form E'v, where v € H, r is a multi- 
index with }7 .<ot Ta = 7, and Ev = THacet Ete. By formulas (2.1) and (2.2) 
it is easy to verify that 

c:H, — Hyy, for x € p_, 
(3.2) c:H, — H, for x €&, 

wv: Hy, = Hn-1 for x € Pi, 
for each nonnegative integer n. The space H _; is understood to be the trivial 


space. From this it follows that @,,.,. Hn is a g-invariant subspace of Hl which 
is dense by irreducibility. 


REMARK 3.1. Since, by unitarity, the adjoint of dw(X) is —dw(X) formulas 
(3.2) imply that the sum @nr>do H, ts an orthogonal direct sum. 


We now define an operator valued multiplier J = J, in terms of 7 and the K 
valued multiplier 7 defined by equation (2.6). Let (g,T) € Gc x pi, for which 
gexp(T) € 2. We define J(g,T) = 7(j(g,T)). Since 7 is a homomorphism, it 
follows from equations (2.7) that 

J(k, T) = r(k), 
(3.3) J(p,T) =I, and 
J(9192,T) = J(91,92-T)J(92,T). 
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4. Three-Dimensional Systems 


We develop some results for representations of three-dimensional systems hav- 
ing highest weight vectors. We assume G is a real Lie group and p is a unitary 
representation of G. If H, E, E are in g, the complexification of the Lie algebra 
of G, and if [E, E] = H, [H, E] = 2E, [H, E] = —2E, then the triple {H, E, E} 
is a three-dimensional system. We say v is a highest weight vector for the three- 
dimensional system {H, E, E} if v is an analytic vector for the representation p, 
dp(H)v = pv and dp(E)v = 0. As before we use the notation Y -w or Yw to 
denote dp(Y )w for C™ vectors w and vectors Y € U(g). 

LEMMA 4.1. Let {H, E, E} be a three-dimensional system. Then 

(a) (H, E") = -2nE". 
(b) [H, E"] =2nE". 
(c) EE” = E°E+nE" 71H —n(n-1)E™}. 
Proor. Note [H, E] = —2E gives HE = EH — 2E. Thus 
HE? = EHE - 2E° 
= E°H — 4B”. 
Hence suppose HE" = EH — 2nE”. Then 
HE™*1=HE"E 
= (E"H —2nE")E 
= E"EH -2E"*! —2nE™! 
= E™+'H -2(n+1)E"". 
This gives (1). A similar argument gives (2). For (3) first note EE = EE + H. 
Hence if EE” = E"E +nE"—1H —n(n—1)E"~! we see 
EE") = (E"E4+nE"'H—n(n-1)E")E 
= E™1B4+ E°H4+nE"A — 2nE” — n(n—-1)E" 
= E™*'E 4+ (n+ 1E"A — (n+1)nE". 





0 


LEMMA 4.2. Suppose v is a highest weight vector with weight u for the three- 


dimensional system {H, E, E}. Then 
(a) E™E™y = CM ee) En—my ifm<n. 
(b) E"E"y = np netaye. 
(c) EM E"y =0 ifm>n. 





PRooF. By Lemma 4.1 EE"y = nE"-!Hv — n(n —1)E"~10 = n(w—nt+ 
1)E"-1y. Suppose 





n! Pius n+k+)) onze, 


EF E™ = 
° (a —k! P—ntl) 
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Then again by Lemma 4.1 


mo T(w-n+k+1) 
(n—k)i Tyw—n+1) 
— mt Tw-n+k+1) eee ie] 
~ (n—k)t T(p~n+1) Peer ny 

—(n—k)(n—k—-1)E"-*"!v) 
— mt T-n+k+1) =n—k—1 
~ (n—k)! TP(w—nt+1) a ee 
8 Re 2) eis 

(n-—k-1)! T(wp—-n+)) : 


EA By EE™*y 














This gives (a) and (b). For (c) note 


C(u +1) 


E™ Ey = E™ "EB" By = E™- | 
T(u—n+ 1) 


v=0. 


5. The Operator qr 


In this section we define a linear operator gy : H — H, for each T € D. 
Initially, it is defined in terms of a power series involving the exponential function 
and the derived representation dw. In the next section gr will be related to the 
operations of the group representation. 


Let T € p,. Define gr : H — H by the formula 





(5.1) qrv= > ote 
n=0 


where v € H. The next theorem shows that the convergence of this series in H 
completely characterizes D. 


THEOREM 5.1. Let v be a nonzero vector in H andT € p,. Then the series 
that defines qru converges in the Hilbert space H if and only if T € D. 


Proor. Let T € py and v € H. To determine the convergence of grv it is 
enough by Remark 3.1 to determine when the sum $>(n!)~?||T”v||? converges. 
Let a, = eae t;E,, :0<t; < oo}. Then each element T of p; can be written 
in the form T = Ad(k)c for some c € ay and k € K. Furthermore each T of D 
can be written in this same form with the restriction that 0 < ¢; < 1, for each j 
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(Corollary 7.17 of [7]). By unitarity we have for such a T 


S palltrall? = Do raps 
n>0 


oso mF 


“Be lle) "1 (k)ull? 


z||(Ad(k)e)"»||? 








where w = 7T(k)v. It follows that we only need to determine for which c € a, 
the series for g,w converges. Furthermore, since each w € H can be written as 
a finite sum of weight vectors in H we may assume w is a unit weight vector. 
Let yu be the weight of w. We now concentrate on evaluating the expression 
|e" w]|? = (-1)"(c"@" w|w). Write c= t:E; , where E; = E,,,1<i<r. Also 
let H; = H,,. Then 


C= Os t,E;)" = x Be 
aj=n 


where t® = t{* ---¢@r and B® = E?’ --- Ber. Thus we get 


tn! = 7 
(c"aulw) = > aa 1° (E° Hw lw). 


jal=n,|B|=n 
Since E,,--- , E, commute and [E;, E;] = 0 if i 4 j it follows that we can write 
E°E® as a product of commuting operators (E@! BE?) ---(E°* E-). Now sup- 


pose a # 3. Then for some index i we have a; > @; and (E! Ef! --. E& EF wl\w) 
= 0 by Lemma 4.2 (c). It now follows that 


(rwlw) =D (Eyer (E* Bwlw). 


Furthermore, by Lemma 4.2 (b) we obtain 
NAN n! azo a a 
(c?e" ww) ae Bay! --- ay! pO (Ay) ~~» pr) (A) (ww) 


are CY" 0 (Hy) wl (Hy) 


where a‘") = a(a—1)---(a—r41) = rete: Putting this together we finally 
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obtain, 


1 
(nt)? 


—1)” 


ile" w||? 





(qew|q-w) 


Mi 


3 
il 
° 


“—~ 


Gagne ero) 
eiima> 


laj=n 


pln) (Fy) «= wl@)(H,) 
al 


[ory p\%) (Hj) 
a;! 


lor) (Hy) + wl) Hr) azn 


or es 


al 


t*¢e 





=i 


(5.2) Ve 


(les?) 


(-1) 


2M 


ia 
It 
an 


ll 
4, 


tl 

ha 
g 
= 


(-H(Ai)) (as) 55 = (lt ge 


Fo(—p( Hi); |ti\) 


ll 
ams 


i 


where a,) = a(a+1)---(atr—l1)= NCEE By [6] the hypergeometric 
function | Fo(r;s) = (1 — s)~” converges whenever |s| < 1. It follows that the 
series for q-w converges whenever 0 < t; < 1. Thus grv € H, for each v € H 
and T € D. 

To complete the proof it suffices to show c € D when q.w converges where 
c € ay and w is a nonzero weight vector in H. Indeed, note if g7w converges for 
some nonzero vector w € H and T € px, then w(k)grw = w(k)qrw(k7!)r(k)w = 
qz-tT(k)w converges and hence we may assume T = c where c € a+. Next 
note if gw converges for some nonzero w € H, then w = >> w; where w; are 
nonzero weight vectors and ||q.w||? = >>; ||q-w:||?. Hence g.w converges for 
some nonzero weight vector w having weight yu. Since the radius of convergence 
of 1 Fo(—p(A,), \ts|?) = (1 — |t:|?) #79 is 1 for w(H;) < 0, equation (5.2) implies 
ce Dif ie the y4(H;) are negative. 

Let (-,-) denote the inner product on the dual t* of t induced by the Killing 
form. For a and @ in t* let ag = tam. Our choices are such that w(.Hi) = p,. 
First consider the case where p = ) is the highest weight for (r,H). Let ¢ be 
defined as in equation (2.2) of [5]. Let z € R and A. be determined by the 
conditions A = A, + 2¢ and (A. + p,71) = 0 where p is half the sum of the 
positive roots (cf. equation (2.3) of [5]). The condition (A, 7) < 0 is equivalent 
to z < p,,. However, since w is a unitary highest weight representation, z is 
bounded from above by the number B(A.) given by Theorem 2.4 and Theorem 
2.8 of [5]. The explicit formula for B(A,) involves weights p..) and fc. which 
are half the sums of the positive compact roots in certain distinguished subroot 


GEOMETRIC REALIZATIONS 21 


systems Q and R of ®. If p, denotes half the sum of the compact roots of ® 
then p. — fc, is half the sum of some positive compact roots for 1 = 1,2. By 
maximality of the root y; we have (a,7) > 0, whenever a € ®t. From this 
it follows that (pc,i)+, < (Pc)y, for each i = 1,2. Thus by Theorem 2.8 of [5] 
B(A.) < 1+ 2(pc)y,. However, Lemma 5.3 of [5] gives z < 1+ 2(p¢)4, = py, in 
the case when A # 0. Hence A,, <0. Now suppose pu is an arbitrary weight in 
(7,H). Then A — yz is a sum of positive compact roots. Again by maximality of 
1 we have p,, <A, <0. 

The strongly orthogonal roots all have the same length, and 7 is the highest 
weight of the irreducible representation of § on p,. Hence for each 7 there is a 
compact Weyl group element o such that o7, = 7;. Since the o permutes the 
weights of (7, H), it follows that u,, = Uoy, = (07h), <0. This completes the 
proof. O 


6. The operator w in terms of gr 


In this section we relate the operations of the group representation w to the 
operator gr. More precisely, we show 


(6.1) w(g)v = qg.oJ(g,0)*~*v for v € H and g EG. 


First we relate gz to the representation p of a real Lie group G having a three- 
dimensional system {H, E, E} with highest weight vector v. 


REMARK 6.1. Suppose {H, E,E} forms a three-dimensional system in the 
complex Lie algebra of a complex Lie group Gc. Then as seen by Lemma 7.7 of 


[7] 
(6.2) exp(t(E + E)) = exp(tanht£) exp(log(cosh t)H) exp(tanhtE). 


Moreover, if v is a highest weight vector with weight y for he three-dimensitonal 
system {H, E, E} then by Theorem 5.1, 





tanh” t - 
dtanht EV = 23 nl E"v 


is a convergent series in the Hilbert space and gives an analytic function of t. 
This suggests that 


= t i 
plexp(t(E + B))v = cosh" t S> 8 * ny 


DEFINITION. Suppose {H, E, E} forms a three-dimensional system and Hv = 
pu, Ev =0. Define cn..(u) by (E+ E)"v = Pg Cn,e(p) EF. 
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REMARK 6.2. We have coo = 1, e190 = 90, €1,1 = 1. Moreover 


(E + E)*v = E°v + EEv + EEv+ E’v 


= Hv+ Ev 
= put E*v 
gives 
C20 =H 
(6.3) c2,1 = 0 and 
C22 = 1. 


LEMMa~ 6.1. One has 


Cn+1,0(H) = Hen,1(H) 
Cn+1,k(H) = Cnk—1(H) + (K+ 1)(u—k)enepi(y), O<k<n 
Cn4i,n(H) = €nn—1(#) 

Cn+1,nt¢i(H) = Cnn(H) = 1. 


Proor. By Lemma 4.2 


(E+ E)"y = (E+ B)S eng Eto 


k=0 
n e n _ 
= > Cn, bE E*y + So ene Beto 
k=0 k=0 
n + n 2 
= a Cn kk( — b+ 1)ER ly + x Cn pb tly 
k=1 k=0 
n-1 7 n+l 
= ys Cn, koi (k + 1)(u — k)B*o + S> Cn,k-1 E*v 
k=0 k=1 
n-1 
= Cry pu + ee {(kK+1)(u— k)enjk4i1 + €njk—1} Ety 
k=1 


a Cruel o + Ca Ee, 
Thus 
€n+1,0 = HEn,1 
Cn+1,k = Cn,k-1 + (K+ 1)(u- k)enk+1 


Cn+1jn = Cnyn-1 


Cn+1,n+1 = Cn,n- 
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PROPOSITION 6.1. Let v be a highest weight vector with weight y for a three- 
dimensional system {H, E, E} relative to a representation p of a real Lie group 
G. Then fort >0 


p(exp(t(E + E))u = (cosh t)# s = eS 


n=0 


= (cosht)"qtannt EU- 


PROOF. We will show 


5 eee = (cosh ey" > Sa tg "y for t > 0. 


ni 
nm i’ 


This will occur if 


pos Bu = ( cosh t)" hey ae, 





iM: 
B15 


Hence we need to show 





°c 600 oo k 
t” - tanh” t — 
y y “ok E*y = (cosht)# y —— E*y, 
n! k! 
k=0 n=k k=0 


Thus for each k it suffices to show 





foe) 
Cn 4t” as t 
(6.4) > a wae = (cosht)” 


n=k 


Set x = cosht. Then e** — 2re' +1 = 0. This gives t = In(x + Vx? -1). 
However, since t > 0, one has the plus sign. Indeed, if t = In(x — Vx? — 1), then 


t_ 1 
e€ = Set Thus 


t= In(c + V2? — 1). 


We now define C,,,/n! to be the power series coefficients of (cosh ¢)# fanh 
Note if « = cosht, then tanht = oe Set 


k 
‘on t = oat k(n? — 1)#/2, 





fx (x) = (cosht)¥ 


Then 


fla) = D0 Cok n(n + va? — 1)" 
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We calculate fj (x) two ways. First 


k 


f(z) = a gh TN ge 1/2 + geht] (g2 1)(#-2)/2 








(k—1)! 
1 1 














el ca oe CS ye ee 
ip —_ fe = ye ee? = ye-D/2 
al (K+ 1)(u—k)fepi(x) + fr-1(2)}- 


We next calculate f{ (x) from the power series. 
“Cn n-1 1 x 
pa) = In(e+ Va {4 
fal) 25m (in =) aoe Tgrat) 
pr (iat (c+Vz =) aa 





wale 


Combining these we obtain 
Sex C. k n-1 
(k+1)(u— k) facts (2) + fe-ale) = > A (Ine + Va? =) 
ior’ (n— 1)! 
These yield recursion formulas for the C’,,. First we rewrite this as 


(k+1)(u—k)fera(z) + fra(z) = > Ses (ny a =n)’. 


n=k-1 
This gives 
Cy—1,k-1 = Chk 
Cr k—-1 = Ceti yk 
(K+ 1)(u—kK)Cr ati t+ Crr-1 = Cr41,4 forn > k+1. 
Now 


(Q 
cosh’ t = Soho yee fo 
n=0 ni 
Hence Coo = 1, Cio = 0, Coo = p. By tee 6.1 and equation 6.3, Cr,, and 
Cn,z satisfy identical recursion relations and agree for k = 0 and n € {0, 1, 2}. 
Hence we have 
Crk = Cn,k 

for allnandk. O 

We now establish the relation between the representation w and the operator 
qr. Let Z € ag be written in the form 


(6.5) Z= Sot B; + E;) 


i=l 
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where £; = E,, are the root vectors of the strongly orthogonal roots y;. Then 
by Lemma 7.11 of [7] one has the decomposition 


exp(Z) = exp(X) exp(H) exp(X) 


(6.6) 2 
= exp(X) exp(—H) exp(X) 


where 
H= S~ log(cosh t;)H; and X = > tanh Ej. 
i=l i=1 
Let v € H be a weight vector with weight yw. Recall that E;v = 0 for each i. 


We set yz; = (H;). Then H;-v = pv for each i. Therefore v is a highest weight 
vector for the three-dimensional systems {H;, F;, Ej} relative to w. 


PROPOSITION 6.2. One has 


w(exp(_ti(E; + E,)))v = | [ (cosht,)" exp(} > tanh t;E,)u 
= [[(cosh oe qs- tanh t; EB, v. 


ProoF. The series on both sides converge. Hence we need only show they 
are equal. Consider the left-hand side 


exp()_ ti(Ei + Ei))v = x oy —(E +E)*v 
k=0|o|=k 


os mies 
where a = (a1, Q2,° re ar) and t® = ty ty? a4 tr and 


(E+ B)* = (E, + £,)™ ---(Ep + E,)™. 


Note 


(Ei + By)" (Bj + By) 0 = (Bit B)™ § YS cay,p, (uy) By 
8;=0 


= > Cai .B; (u;) EY? (BE, + E;)*v 
B,=0 


Oj, OF 


: d. a Cas8; (Hi) Ca;,p; (uj) EP BF v. 


f.=0 B;=0 


Hence define ¢o,a() = T]ca;.a,(ui) where 8 < a. Set B® = Ef... Bo. We 
thus see 
(E+ E)?v= S- co,e(u) BP. 


BSa 
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Hence 


exp(S_ (Ei + B))o= > > at? Yo ca,o(u) Bw 


k=Olal|=k ~°  B<a 


= 1 
=S°>d> 5 it ca,a(u)EPv 
k=0|6|<k a>6 
la|=k 


Now the right hand side can be rewritten in a similar way. 


A¢ 


[[ (cosh t;)¥ exp()_ tanht;E;)v = [[(cosh ty)" S- a(S tanh t;E;)*v 


k=0 


= = [[(cosht,)" ae 5 tanh t)? Ey 


=0 ajar? 


Ag 


where (tanht)? = [| tanh Pe Hence we have equality if their projections onto 
HH, are equal. Hence we need to show 


3 eS A t%¢a,a( )E*v = | [(cosht;)"* S> = A + (tant)? Ev. 


k=0 |6|= gece [B|=l 


But this follows if 





- Ca (HL) po =F ~ T] (cosh ti) “(tanh t)?, 


a! 
a2p 
This however is implied by (6.4). O 


THEOREM 6.1. Letv € H andg€G. Then 
w(g)v = qo-0J(g,0)*~1v. 


PROOF. It is enough to assume that v is a weight vector in H with weight 
pL. To establish the formula for a € A it suffices by Proposition 6.2 to note that 
a-0 = m4(a) = Sotanht; EF; and J(a,0)*"!v = J[](cosh t;)4'v, when a = 
exp()7j- t;(E; + E;)). By equation (6.6) we can write a = exp(X) exp(—H) 
exp(X), here He 7) log(cosh(t;))H; and X = S7;_, tanht; E;. Thus a-0 = 
>o, tanh t; E; and 


J(a,0)*~!v = T(10(a))*~'v = T(exp(—H))*~!v = r(exp(H))v = [[(cosh t,)Miv 
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For general g € G, we can write g = k,ak2 where k,,ko € K. Then 


= QAd(ky)(aka-0)J (K1, akg -0)*~* J (ak2,0)*~*v 
= Qk, ak,-04 (ki ake, 0)*-!y 
= I-04 (9, Oi: 


7. Intertwining Operators and Reproducing Kernels 


The operator gry will play a decisive role in determining the intertwining op- 
erator between the given realization (w,H), and the standard realization of the 
highest weight representation as a space of H-valued functions on D. Further- 
more, qr will be used to determine the reproducing kernel for the standard 
realization. 


PROPOSITION 7.1. LetgeG,T ED andve H. Then 


w(g)arv = qg-7rJ(g,T)* ‘v. 


PRooF. Choose g; so that g; -0 = T. Then by Theorem 6.1 


w(g)arv = w(g)w(g1)J (gi, 0)*v 
= w(g91)J(g1,0)*v 
= Ggq1-0F(991,0)**J(g1,0)*v 
= 49-7 J(g, 91 -0)*~* (91, 0)*-* J(91, 0)*v 
= q9-7J(g,T)**v. 


Oo 


PROPOSITION 7.2. The span of the vectors qrv, T € D, v € H is dense in 
the Hilbert space H. 


Proor. This follows from Proposition 7.1 and the irreducibility of w. O 
LEMMA 7.1. For vectors v,w € H, we have (qrv\w) = (v\w). 


Proor. Let T € D and v € H. Then by equation (3.2) grv = exp(T)v = 
u+v’, where v’ € @i5,H;. By Remark 3.1 v’ is orthogonal to H = Ho. The 
result now follows. O 
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THEOREM 7.1. There is a positive definite operator-valued function Q :D x 
D — Aut(H,) such that 
(arv|qsw) = (Q(S,T)v|w). 


The function Q is holomorphic in the variable S and antiholomorphic in the 
variable T. 


Proor. Choose g; and go so that g,;-0 = T and g2-0 = S. Then Proposition 
7.1 gives 
qrv = w(g1) J (gi, 0)*v and 
qsw = w(g2)J(g2,0)*w. 
Then 


= (w(g3 91) J(91,0)*v| J(g2, 0)*w) 

= (dg=14,.0F(92 91,0)**I(gu, 0)* >| J (ga, 0)*w) 

= (J(g5', 91 -0)*~tv|J(g2,0)*w) by Lemma 7.1 

= (J(g2,0)J(g7',T)*~*v|w). 

Now write go = 7+(g2)™.(g2)m-(g2). In the following let o : Gc — Ge denote 


the real Lie group homomorphism whose differential is conjugation on g. Then 
we have 


(qrv|qsw) 


92" = 0(g2)* = om _(g2)~* 0M 0(g2) om 4 (92)? 
From (2.6) and the definition of J(g,T), we easily obtain 


(7.1) Jo; Ty =J(g Ty i-ae Pt 


(7.2) J(kg,T) = 7(k)J(g,T) if k € Ke. 


From this we get 


= 1(to(92))J(om—(g2)~* 7092)", 074 (g2)* - T)** Ions (g2)",T) 
= 7(to(92))T(oM0(g2)~*)** J(om4 (92) *, 7)" 
= J(exp(—S$),T)*7?. 


It follows now that Q(S,T) = J(exp(—S),T)*~! and Q(S,T) is holomorphic in 
S and conjugate holomorphic in T. O 


PROPOSITION 7.3. The reproducing kernel Q is positive definite. 


PROOF. By Proposition 7.1 


SQ Tvs) = (Do anvil Yo ares) > 0. 


24J 
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Since Q(-,-) is a positive definite operator valued kernel we can follow Kunze 
[8] to construct a Hilbert space H(D, 17) of continuous functions f : D — H with 
the following properties: 

(1) Thespan of the set {S++ Q(S,T)v : T € D, v € H} is dense in H(D, 7); 

(2) For S € D, the map es : f — f(S) is a continuous map from H(D,7) to 

A, 

(3) Q(S,T) = ese%, for all S,T € D; 

(4) (Q(-, T)v|Q(., S)w) = (Q(S, T)v|w), for all S,T € D and v,w,€ H. 
When 7 is singular, the space H(D,r) is a proper subspace of Hol(D,7) and 
can be described as the solutions to certain differential operators. We refer the 
reader to [3, 4] for more details. 


PROPOSITION 7.4. The formula 
T(g9)f(S) = J(g7", S)"*F(g7* 5), 


where f € H(D, 1), defines a strongly continuous unitary representation of G on 
H(D,r). 


PROOF. This is easily verified. See [8] for details. O 
PROPOSITION 7.5. Let S,T € D and letg EG. Then 


PROOF. By the definition of J one can easily verify that e,.r = J(g,T)er 
T(g~'). From this we get 


Q(9°S,9°T) =e9.5 5.7 
=J(9,S)esT(g-')T(g"*)* ef J(g,T)* 
=J(g, 5) esey J(g,T)* 
=J(9,5) Q(S,T) J(g,T)". 
oO 


THEOREM 7.2. The mapping grv +> Q(-,T)v extends to an intertwining op- 
erator = between (w, HI) and the geometric representation (T,H(D,H)). It can 
be defined globally by =f(S) = qi f for f € H. 


PrRoor. Let v; € V and T; € D. By the definition of the inner product on 
H(D,7) and by Theorem 7.1 we have 


WD QC, Teall? = > QUT, Tviles) 
= Do (arvilar,es) 
=| Deanvill?. 
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By Proposition 7.2 it follows that the map grv — Q(-,T)v is well-defined and 
extends to a unitary map =: H - H(D,r). Let g € G and let T € D. Then we 
have by Propositions 7.1 and 7.5 


E(w(g)arv) =E(dgrJ(g, T)*~'v) 
=Q(.,9T)J(9,T)* 10 
=J(g™",-)Q(g7"(-), T)v 
=T(g)(Q(-,T)v) 
=T (9)E(qrv). 


It now follows that = is a unitary operator intertwining (w, H) and (7, H(D, r)). 
C 


8. An Example 


As an example and application we consider the harmonic representation w. 
Discussions of this representation appear in many articles in the literature. We 
will adopt much of the notation and results stated in Chapter 7 of [4]. Let 
G = SU(p,q), Sp(n,R), or SO*(2m). The Hilbert space for the action of G is 
described as follows: We let M be the complex vector space which depends on 
G and is described by (7.2) of [4]. Then the Hilbert space is the Fock space F 
consisting of all those holomorphic functions on M which are square integrable 
with respect to the Gaussian measure. Let 6: M — p, be the map defined 
there and Z the ideal generated by the matrix entries of 9 and the constants. 
If A is an irreducible K-space of harmonic polynomials then the closure H of 
TH is an irreducible subspace of F for w. On p, we let (-|-) denote the standard 
complex inner product induced by the Killing form. Then for x € p, the action 
of Z is given by Z- f(z) = (6(z)|z) f(z). From this it follows that for J € D the 
operator gr is the multiplication operator 


(8.1) arh(z) = eC) h(z) 

where h € H. The intertwining operator is determined by the formula 

(8.2) (S(T) |h) = (¢larh), 

where ¢ € H, h € H, and T € D. If K(.,-) is the reproducing kernel on H then 
the intertwining operator is given explicitly by the formula: 


(8.3) E¢(L,z) = / b(w)eT)) Kw, z)d(w). 
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In [1], we studied automorphism groups G of Siegel domains © and their 
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of G is the symmetry of © with respect to the point ie. Realized in a vector- 
valued Schrédinger-Fock Hilbert space GG, the discrete series S,, of G at 7 is the 
Hankel transform K,. In [I], we defined K,, as an integral operator, the kernel 
of which is the Bessel function K,,, and we showed that the square-integrability 
of K,, implies the square-integrability of the discrete series of G. This approach 
generates the theory developed in [6], [7], [2]. In this paper, we compute K, 
explicitly on a dense subspace of Gp. Using this new description, we prove the 
square-integrability of S,. This new approach to studying the discrete series of 
G is independent of Bessel functions. 
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Let (V,V~) be a simple Jordan pair with a positive Hermitian involution 
x++ £. With respect to a maximal tripotent e, V is decomposed as V = Vo @ Vj. 
Then V, is the complexification of a formally real Jordan algebra J, in which 
the interior 2 of the set of squares is the positive cone. With a positive definite 
bilinear map B: V; x V; — V2 we can define a Siegel domain © of type II by 


(1) JE (eae we vitine= 5 Blu, > 0}. 


Let A be the Koecher norm function on 2, r be the rank of 2, n, and n be the 
dimensions of V2 and V over C, respectively. Then n = n; +rb for some positive 
integer b. Denote 4 = *, which is an integer or a half-integer. The Cayley 
transform c is a biholomorphic isomorphism of a circled bounded symmetric 
domain D with D. As in [1], we define a trilinear map { }: Vx V~- x VV 
and define 


(2) roy = {xey} 


where z,yE V. 

Let G’ and G be the groups of biholomorphic automorphisms of D and ©, 
respectively, K be the maximal compact subgroup of G’, and K© the complex- 
ification of K. As in [3], we decompose the Lie algebra g of G as a canonical 
graduation 


e 


(3) g9=9-189_1 O90 O91 Oni, 


and denote by G_, G_i 
groups. 

Viewing G as an automorphism group of the domain © given by (1), Go, G 4, 
and G; act on © as follows. 

(a) Elements a in Go can be described as 


, Go, Gi, and G, the corresponding analytic sub- 


(4) a-(z,u) = (az, au). 


Le., as an operator on V2, a € GL(V2) and aQ = 2; as an operator on Vi, a € 
GL(V,). Moreover, 


(5) a(B(u, a)) = Blau, ai). 


(b) Elements in G 1 can be parametrized by elements uo in Vj; i.e., for up € Vi 


(6) R(ug) -(z,u) = (2 +1 B(u, to) + 5 B(uo, io), + ug). 

(c) Elements in G; can be parametrized by elements xo in J; i.e., for zo € J 
(7) R(zo) - (z,u) = (+ Zo, u). 

We also study the element 7 given by 
1 


(8) j+ (2, u) = (-277,-27% ou). 
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Let A be the connected component of the identity in Go, and L the maximal 
compact subgroup of A. We identify Vi with a subgroup K = of K© by the map 
ug + k(uo), where k(uo) acts on V2 @ Vy by k(ug)- (z,u) = (z,z0uo +u). Note 
that as subgroups of K©, A normalizes K_ i, and AK_. is a subgroup of K c 

Let 7 be a finite-dimensional irreducible holomorphic representation of K, 
denote by V = V, the space on which 7 acts and by € = €, the space of all linear 
transformations on V. Let M = (mj,mz2,---,m,) with m; > m2 >--: > m, 
be the signature of the restricted lowest weight of 7, and set w(7) = m,. We 
will study V-valued or €-valued measurable functions f on AK ea, which are 
holomorphic in ug € V;. f is called 7-covariant if 


(9) f(ak(uo)l) = m(l)~* f(a k(uo)) 
for alla € A, up €VYj, andl eé L. 


We recall from [1] that the €-valued gamma function [ on © is defined by 


Pe u), a, T) > i e7terle) e2(euluo) e7 (volvo) 


(10) AK_4 


x m((ak(iug))* ak(iug))~* A(ae)* da dug 


for a € C and (z,u) € ©. When w(7) > wp — Rea — 1, the integral (10) defining 
the gamma function converges absolutely. We set 


(11) T,.(a) =T(($,0),0,7). 


In this paper, we assume 


(12) w(t) > 2n+6-1. 
Then 

(13) Pr, =I ,(-p— 6) 
is defined. 


Let G3,(AK_1) be the space of all V-valued m-covariant measurable functions 


fon AK = which are holomorphic in wg and such that 


(14) iz I|f (a k(ug)) ||? e 2!) da dug < oo. 


Nie 


6S, (AK 4} is a Hilbert space with respect to an inner product, 
5) (fille) =f) (afalah(uo))]felak(uo))) €°29) da du 
Bk 
~ 2 


and we denote the resulting Hilbert space by Gfrp. 
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For any f € G3r, 
(16) F(z,u) 


n 


=n7 2 i ef(7l2e) e(ulauo) e~(uoluo) (@k(itg))* f(ak(ug)) da dug 
AK_1 


is absolutely convergent for all (z,u) € © and defines a holomorphic function 
F:% + Y,. We think of F as the Laplace transform of f. Denote by H, 
the space of all functions F on 5 of the form (16) for f € G§,. Let P be the 
quadratic representation of the Jordan algebra V2 = J©, and for zp € Vo define 
a mapping r(zo) on V2 @ Vi by 


(zo) : (z,u)  (P(29)z, zo 04). 


Then H,, is a Hilbert space with the inner product given by 
(F|G) = ‘| r{ G(x + iy, u)* r(k(iu)) r(r(2y — B(u, a)))~! 
(17) 2 
x m(k(iu))* F(x + iy, u) Ay — B(u,i))~2#-* de dy du, 


and the mapping T : f +» F given by (16) is a unitary map of Gp onto H,. 
Moreover, H,, is the space of all holomorphic functions F : © — VY, satisfying 


[ II (r(2y — Blu, a)? k(—éu))* F(a + iy, u)|[? 


Nie 


x A(2y — B(u,ii))~2"—° dr dy du < ow. 


Elements a in Go, R(ug) in G hy R(zo) in Gi, and the element j generate the 
group G. For these generators the unitary representation T;, of G acts on H, as 
follows 


(T,, (a) F)(z, u) = (a) F(a-'z,a7' u), 
(T,(R(uo)) F)(z,u) = m(k(—iuo))* F(z — 7 Blu, to) 
(18) + 4 B(up, tio), u — uo), 
(T,(R(xo)) F)(z, u) = F(z — 20, u), 
(T(j) F)(z, u) = (r(z))m(k(u)) F(-271, 277 ou). 
By (16) and (18), S, = 7~'T,T is a unitary representation of G on G§p given 


ek 


(Qo) f)(ak(u)) = flap ak(u)), 
(ak(u)) = e~ 3(2° ola” uo) e(ula” 40) F(a k(u — a* up), 
(ak(u)) = e*@ola? F(ak(u)), 
(u)) = (Kr f)(ak(u)), 
where K.,, = T—'!T,,(j)T is called the Hankel transform on G§p. In [1], Kx is an 


integral operator, the kernel of which is the Bessel function K, on AK_ 1. We 
now compute K,, explicitly on a dense subspace of GF. 
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For (z’,u’) € &, we define an €-valued function f(,/7)(-) by 


(20) feuy(ak(uo)) = 17? e~Heel2) elarolu) D1 ar (ak(iug))}. 


WT 


It is clear that. f(,/ ..”)(-)v satisfies (9). By (10) and (12), f(./.4/)(-)v also satisfies 
(14). Thus, f(z’ u/)(-)u € G&p for every v € V. 


THEOREM 1. The functions fi: y)(-)v for (z’,u’) € & and v € V span a 
dense subspace of Gr. 


PROOF. By (15), (16) and (20), for F = Tf and v € V we have 
(F(z',u’)|v) = (fl feu ()v). 
Since 7 is a unitary map of G¥p onto H,, 
(21) (F(2',u’)\v) = (Fl Foerwy(-)e), 


where F(z ur) = Tf(z7,ur; ie, F(z uy is the reproducing kernel of H,. If F € H, 
is orthogonal to Fi./4/)(-)v for all (z’,u’) € © and v € V, then F(z’, wu’) is 
orthogonal to all v € V by (21). Hence F(z’, wu’) = 0 for all (2’,u’) € & and 
F=0in#H,. Thus {F(.-y/)(-)v : (2’,u’) € &,v € V} span a dense subspace of 
H,, and {f,2/,u/)(-)u : (2’,u’) € Xu € V} span also a dense subspace of G¥p. 


THEOREM 2. For (z’,u’)€ 5 andv ey, 
(22) Ke fer ay 0 = fcart-atouy() t(k(w))* a(2!)*0 
for all fiz uy(-)u € Gr given by (20). 
PROOF. By (17), 
((Tr(9))*F)(z, u) = 1 (2) w(k(u)) F(-277, -27 ou). 
By (21), 
(Tn (9) ) Fer ury()¥ = Feat, -2' tour) (+) w(R(w’))* mz’) *0. 


Since Fiz/u) = T fiz) and K, = T~'T,,(j)T, we have (22). 

By Theorem 1, (22) gives an explicit form of the Hankel transform K, on a 
dense subspace of G§r. 

We now adopt a lemma from [1]. 


LEMMA 1. (cf. [1], Lemma 5.5) For u € Vi and invertible z € V2, we have 
k( — u)r(z)~! k(—éu)* r(2Im z — B(u, %)) k(—iu) r(z*)~! k(—u)* 
= k(-iz7! ou)*r(2Im (—z7') — B(z7' ou, z-lou)) k(-iz} ou). 
It follows from (23) that 
k(iu)* r(z) k(u) k(—iz! 0 u)* r(2Im (—271) — B(z71 ou, z-1 0n)) 
x k(-iz7! ou) k(u)* r(z*) k(iu) = r(2Imz — Bu, a)). 


(23) 


(24) 


We now show the square-integrability of S, by the new description of K,,. 
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Let dz and du be Lebesgue measures on J and V,, and da Haar measure on 
A. R(uo) and R(x) commute, and generate a subgroup N of G. Since M = NA 
is a semidirect product, 


(25) a f(m)dm = i: = f(R(xo) R(uo) a) dtp dug da 


defines a right Haar measure on M, and this equality is valid for any nonnegative 
measurable function f. Moreover, the formula 


es) ft peodg= ff sj (20) R(uo)) dm azo duo 
G M JJOV; 
is valid for all nonnegative measurable functions f. 
LEMMA 2. (cf. [1], Theorem 7.8) For any f; and fo in G§r, 
J sem) filfayPam <2 0" fu? 
M 


27 ; 
. ‘ Ce. |IT'# fo(ak(uo))|I? eM? A(ae)-#” da duo). 


THEOREM 3. For any f; and fy in Gp, 
(28) J Se@hilsa)? da < 00. 
G 
Proor. By (26) and (27), 
[is 9) filfe)|? dg 
= ff MSelm)fals-( R(uo) R(eo)) fi)? dm dy du 
J@V, 


(29) " 
< 2" 0" || fall? a (f.. [PF (Sr R(uo) R(20)) f:)(ak(u))||? 


ui 


sce ule) A(ae)~"~° da au) dzo duo. 
We assume temporarily that f; = f,2/u/)(-)v for some (2’,u’) € L, v € V. By 
(19) and (22), we have 


Sir(j R(uo) R(®o) f(z,w))(@ k(u)) v 
aus =? eo ilael—(xot2' +} B(uo,tto) +i B(u',tio))~") 
(30) x e(eul—(o+2'+ § B(uo ,tio )+4B(u' ,tio))~*o(uotu’)) ig n(a k(iu))~! 
x 1(k(uo + u’))* r(x9 + 2’ + 5 B(uo, to) + iB(u’,to))* m(k(iuo)) v. 


Denote 


(31) “=a t2' + 5 B(uo, to) + 1B(u', tig) 
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and 
(32) uy =uo tu’. 


Applying variable changes, (10), (11) and (13), we have 
J Cfo, Wek 2G Rtu0) R(e0)) fen) (aH) oP 
J@eVi AK_} 


x e l4) A(ae)-#~? da au) dxo dug 


<a? [Ps |? (C2 [ler (R(—ée’) II? 


x A(-2Im 27! — B(zyj! ow, zp! ou)? 
JoV: 


x |[a(r(—2Imz_! — B(zy' om, 27" 0 u))2k(—izy! 0 U1) k(u)*r(2t)k (iu) ||? 


x dxo dug. 
By (31), (32) and a calculation, we have 
2Im 2, — B(u,%) = 2Imz’ — Bu’, Ww’) 
and 
~2Im2{! — B(zj! ou, 2, 10m) = 2) (2Imz’ — Bw’, w’)) 271. 


It follows now from (24) that 
i ( J ltd (eG Revo) R20) fern abu) oP 
Jovi \JaK_y 


e(4l%) A(ae)-#—° da au) dzq dug 


33 ; 1 
a Smo" (Px? ||? WF |P le (e(—aw’)) II? IIe (r(20m 2! — Blu’, @'))?) v|]? 


x A(2Im 2’ — B(u’,a’))#t? 


xf 
JeVi 


By [5, (5.98)], 


i 


By (29), (33) and the fact that (z’,u’) € L, (28) holds for f; = fzw)(-)v and 
for all fo € GBr. Since the functions f(.)4/)(-)v for (z’,u’) © L, v € V span 
a dense subspace (G§r)° of Gp, it follows from Fatou’s lemma that (28) also 
holds for all f,, fo € G&¥r, which completes the proof of the theorem. 


mon 


A(z ta’ + 5 B(wo, to) + PBY Mo) dxo dug. 





—2p—2b 


A(to + 2’ + 5 B(uo, to) + iB(u', to)) dx dup < 00. 
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Fonctions Sphériques sur un Espace 
Symétrique Ordonné de Type Cayley 


JACQUES FARAUT 


en hommage a@ Ray Kunze, 4 l’occasion de son 65-iéme anniversaire 


ABSTRACT. For a special class of ordered symmetric spaces, called sym- 
metric spaces of Cayley type, we give a formula for the spherical functions 
by using Jordan algebras methods, which is more explicit than in the gen- 
eral case, and we compute the function cp(A) occuring in the asymptotic 
behaviour of the spherical functions. 


1. Introduction 


Une structure causale sur une variété M est la donnée d’un champ de cénes 
C, sur M. Pour tout x € M, C, est un cone régulier dans l’espace tangent 
T,(M), et C, dépend différentiablement de x (un cone régulier est un cone 
convexe, fermé, pointu et d’intérieur non vide). Une courbe ¥ : [a, 3] + M de 
classe C' est dite causale si, pour tout t, +(t) € C4). S’il n’existe pas de courbe 
causale fermée non triviale la structure causale est dite globale. Elle définit alors 
un ordre sur M : on note x < y s’il existe une courbe causale allant de z a y. 
Si les intervalles 

D(z,y)={zEM|zez<y} 
sont compacts, la variété causale M est dite globalement hyperbolique. 

Supposons que M soit un espace homogéne, M = G/H, ct G est un groupe de 
Lie connexe et H un sous-groupe fermé. La structure causale est dite invariante 
si 


Cr(g)e = dr(g)z(Cz), 
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ot 7(g) est la transformation rH +> grH. Soit mo = eH le point de base. Une 
structure causale invariante sur M = G/H est déterminée par la donnée d’un 
cone régulier C' dans T,,,,(M) qui est invariant par H. 

Supposons de plus que M = G/H soit un espace symétrique : il existe un 
automorphisme involutif o de G tel que 


(G7)o CH CG’, 


ott G? = {g € G| o(g) = g} et (G7)o est la composante connexe de |’élément 
neutre dans G’. Notons g et § les algébres de Lie de G et H, 


h = {X €g|o(X)= xX} 


(nous notons également o la différentielle de l’automorphisme involutif a). Po- 
sons 

q = {X €g| o(X) =-X} ~T,,.(M). 
Ainsi une structure causale invariante sur M = G/H est déterminée par un cOne 
régulier C C q qui est invariant par Ad(#). 

Nous supposons que G est semi-simple de centre fini, et que (g,) est une 
paire symétrique irréductible. I] existe une involution de Cartan # qui commutte 
avec 0, 700=90o00. Soit 

g=t+p 
la décomposition de Cartan correspondante. Si CN t = {0}, la structure causale 
invariante correspondante est globale, et définit un ordre sur M pour lequel M 
est globalement hyperbolique. On dit que M est un espace symétrique ordonné. 
Posons 
S={gEG|g-mo = mo}. 
Alors S est un semi-groupe fermé et S = exp(C)H ((2], Theorem 4.1). 

Il existe un sous-espace de Cartan a C p qui est contenu dans pMq. Soit 
A = A(g,a) le systéme de racines associé. On choisit dans A un systéme positif 
At (voir [2], section 3) et on pose 


n= Ly 3°, p=s x Ma Q, 


acAt acAt 


A=expa, N =expn. 
L’ensemble NAH est un ouvert de G. Pour x € NAH, on pose 
x = nexp A(z)jh. 


On montre que S C NAH ([2], Corollary 3.2). La fonction sphérique y (A € a*) 
de l’espace symétrique M = G/H est définie sur l’intérieur 5° de S par 


=) a dh. 
H 
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Il existe un élément Xo tel que les valeurs propres de ad(Xo) soient 1,0,-1. On 
note 


Ag = {a € At | a(X) = 0}, 

A, = {ae A]a(Xo) =1 c At, 

S> at, m= Sg, 

acat acai 

No = expno, Ni =expny, Ni = 6(M,). 


Ainsi At = At U Ay, et N = NoN, est le produit semi-direct de No et de N,. 
L’ensemble D = N, 1 NAH est un ouvert bormé de N,. L’intégrale qui définit 
Y, converge pour tout z de S° si et seulement si 


No 


cp(A) =| elPtA,A(A1)) dn, < 00 
D 


([2], Theorem 6.3, od cette intégrale est notée c_(A)). Les fonctions sphériques 
y définissent des caractéres de l’algebre de Volterra V(M)* et interviennent 
dans la transformation de Laplace sphérique (voir [2], section 8). 

Dans cet article nous considérons une famille d’espaces symétriques ordonnés, 
appellés espaces symétriques ordonnés de type Cayley ((8], Definition 5.7). La 
géométrie et l’analyse de ces espaces peuvent étre étudiées a l’aide des algeébres de 
Jordan (voir [6] et [7]). Nous explicitons la représentation intégrale des fonctions 
sphériques y, et nous calculons la fonction cp(A). 


2. L’espace symétrique ordonné SL(2,R)/SO(1, 1) 
L’espace M = SL(2,R)/SO(1,1) est exemple le plus simple d’un espace 
symétrique ordonné de type Cayley. I] peut étre réalisé comme 


M = {(z,y) € Pi(R) x Pi(R) | c 4 y}, 


le groupe G = SL(2,R) agissant sur M diagonalement. Le sous-groupe H = 
SO(1, 1) est le sous-groupe d’isotropie du point mp = (1, -1). Sia est l’automor- 
phisme involutif de G défini par o(g) = JgJ avec 


01 
7=(3 9): 


H =G" = {g€G| o(g) = 9}. 
Ainsi M est un espace symétrique. Notons g et 9 les algébres de Lie de G et H, 


alors 


p= (x ea] o(x)=x}={ (4 3) [zeR}. 


Posons 
q={X €g|o(X) =—-X}, 
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alors 


Soit C Cc q le cone défini par 


o={(", Jutv<0, u-v>oh. 


Ce céne C est invariant par AdH. II définit une structure causale invariante sur 
M, et fait de M un espace symétrique ordonné globalement hyperbolique. Nous 
choisisons a = RX, at = Rt Xo, avec 


Alors At = {a}, avec a(tXq) = t, et 
A={a=(4 vs) [teR} 
n= {ne=(j 1) wer}: 


NA-mpo = {(e' + u,—e* +u) | t,u€ R} 
s’identifie au demi-plan {(u,v) | v > 0} (v =e). Considérons sur W = R? la 
structure d’algébre définie par 


L’orbite 


(u1, 01) (ue, V2) = (urug + U1 v2, U1V2 + U2). 


Identifions 4 R la sous-algébre {(u, 0) | u € R} de W, et posons j = (0,1). Alors 
j? =1et (u,v) =u+ jv. Cette algébre est isomorphe a l’algébre produit R x R, 
Visomorphisme étant défini par 


w=utjurh (utv,u—v). 


Un élément w = u+ ju de W est inversible si u2 — v2 £0. Si 


a b 
g= (: i) €G= SL(2,R) 


on pose 
g-w=(awtb)(ew+d)", 
linverse étant pris au sens de l’algébre W (g- w n’est défini que si cw + d est 
inversible). Cette action de G = SL(2,R) correspond a ]’action de G sur l’espace 
symétrique M. 
Par analogie avec le corps des nombres complexes nous noterons, si w = u+Jju, 


Rw =u, Sw=v. 
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Siw = ut ju = nya; - mo, alors t = log(Sw), et A(w) = log(Sw) - Xo. Pour 
Rs > 0 soit =, la fonction définie sur 


S°.mo = {(u,v) |O<v <1, ju] <1-—v} 


par 
2,(w) = [ee -w))*dh. 


Alors p,_p(w) = &,(w), si (Xo) = —A1, $= A1 +}. Un élément de H s’écrit 


et un calcule simple donne 


6 6 \-1 
Oh . 4a) — yale —o (eee 2" 
S(h- jv) = (ch he sh 5) : 


Par suite, pour 0 <v <1, 


E5(jv) =| (cn7S0- - sh?Sv) "0. 


Cette intégrale converge pour Rs > 0. Elle peut s’écrire 
co 
=.(je') = / (cht + ch@sht)~°dé. 
—ooO 


La fonction =, s’exprime a l’aide d’une fonction de Legendre de seconde espéce 
: pour t > 0, 

=.(je-*) = 2Q,_1 (cht). 
et 


pale!) =2Q, 1 (cht). 


i 
2 


m= N= {a= (1 :) | wer}, 


et S(mu- 7) = (1—u?)~! > 0 lorsque -1 < u< 1. On en déduit que 


Nous avons 


D=NONAH = {’i,|-1<u< 1}, 


et 


1 1 
en() = f (1) Fdu= BOs + 4,4), 


ou B désigne la fonction beta d’Euler. 
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3. Cones symétriques et algébres de Jordan 


Soit 2 un céne ouvert convexe dans un espace vectoriel euclidien V de dimen- 
sion n. On suppose que 2 est pointu. Soit G(Q) le groupe des transformations 
linéaires qui préservent 2, 


G(Q) = {9 € GL{V) | gQ = O}. 


Le cone 2 est dit homogéne si G(Q) opére transitivement sur 2. Le céne ouvert 
dual 0* est défini par 


O* = {rEeV|Vye 2\ {0}, (xy) > O}. 


Le cone 2 est dit autodual si O* = 2. S’il est homogéne et autodual, le cbne 0 
est dit symétrique. 
Une algébre de Jordan euclidienne est un espace vectoriel euclidien V muni 
d’un produit vérifiant 
(1) xsy=~yz, 
(2) 2(x*y) = 2?(zy), 
(3) (xylz) = (y|zz). 
Koecher [5] et Vinberg [9] ont montré que l’intérieur du céne des carrés d’une 
algébre de Jordan euclidienne est un céne symétrique, et que tout céne symé- 
trique peut étre obtenu de cette facon. 
Par exemple si V = Sym(m,R) est l’espace des matrices symétriques réelles 
m xX m muni du produit de Jordan 


roy = 3 (ry+ ya), 
alors V est une algebre de Jordan euclidienne relativement au produit scalaire 
défini par 
(zly) = tr(zy), 
et le céne symétrique 2 est le cone des matrices symétriques définies positives. 
Pour x € V notons L(x) et P(x) les endomorphismes autoadjoints de V définis 

par 

L(x)y = xy, 

P(a) = 2L(x)* — L(2”). 
Si V = Sym(m,R), P(x)y = xyx. Pour x dans 2, P(x) est défini positif. 
Considérons sur 2 la structure riemanienne invariante définie par 

G,(u,v) = (P(z“")ulv), EO, uve V. 

Cette structure riemannienne est invariante par G(Q2), et fait de 2 un espace 
riemannien symétrique qui s’identifie a l’espace quotient Go/Ko ou Go est la 
composante connexe neutre de G(), et 


Ko = {9g € Go | ge = ¢}, 


e désignant |’élément neutre de V. 
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Nous supposons que l’algébre de Jordan V est simple, c’est a dire qu’elle 
n’a pas d’idéal non trivial. Soit r le rang de V et soit {c),... ,c,} un systéme 
complet d’idempotents primitifs deux 4 deux orthogonaux. Tout élément x de 
V s’écrit - 

z=kY- djc;, ke Ko, 45 ER. 

j=l 
Le déterminant A est défini par A(x) = [] Aj, et la trace par tr(z) = > A,;. A est 
un polyné6me homogene de degré r, et on peut supposer que le ae scalaire 
est défini par (z|y) = tr(ry). Si c est un idempotent, l’espace V(c, 1) = {x € V | 
cx = x} est une sous-algébre de V, et pour 1 < j <r, V; = V(c1 +--+ +6;,1) est 
une sous-algébre de V de rang j. Le déterminant mineur principal A, est défini 
par 

A;(z) = Ay, (2), 
ott Ay, désigne le déteminant relatif 4 la sous-algébre V;, et az) est la projection 
orthogonale de z sur V;. Les mineurs principaux sont positifs sur 9. Pour 
s = (s1,...,8,) € C’, et pour x € N, on pose 


Ag (a) = Ar (a)*~82Ao(x)92-83.. A, (x). 


Les fonctions sphériques ©, du c6ne symétrique 2. = Go/Ko sont données par la 
représentation intégrale suivante 


®,(x) = As(kax)dk. 
Ko 
Au sujet de la géométrie et de l’analyse des cones symétriques on peut consulter 


(3). 


4. Espaces symétriques ordonnés de type Cayley 


Comme dans la section 3, soit 2 le c6ne symétrique associé 4 une algebre de 
Jordan simple euclidienne V, et soit Ty le tube dans l’espace complexifié V© de 
base 2, 

Ta =V4iN={z=rt+ty|reV, ye}. 
Le groupe G des automorphismes holomorphes de Tp est engendré par le groupe 
affine P des transformations 


zrsgzta (gEG(), aeV), 
et par la ‘symétrie’ s, 
s(z) =— 

L’espace homogéne M = G/P est compact, et l’application 

V—M, 

ar gP, 
avec 

g(z) = s(z) +a, 
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définit un plongement de V dans M d’image dense. Ainsi M est une com- 
pactification de V (parfois appelée compactification conforme). L’espace M est 
isomorphe a la frontiére de Chilov de la réalisation bornée D de Ty. Le groupe G 
opére sur M, et donc sur M x M diagonalement. Soit M V’orbite de mg = (e, —e) 
dans M x M. On peut montrer que 


MN(V x V)={(z,y) eV x V | A(z—y) £0} 


(voir [6], lemme IV.4.1, et [7]). Soit H le stabilisateur de mg = (e,~e) dans 
G. La paire (G,H) est une paire symétrique relativement 4 l’automorphisme 
involutif ¢ de G défini par 

a(g) =vogoy, 
ou vy est la conjugaison de Ty définie par 


v(z) =z. 


Par exemple, si V = Sym(m, R), alors M ~ Sp(m, R)/GL(m,R). 
L’algébre de Lie g de G est l’algébre des champs de vecteurs X sur V de la 
forme 
X(z) =ut+Tz— P(z)v, 
avec u,v € V, T € go, l’algebre de Lie de Go. Ainsi un élément X de g est un 
triplet (u,T,v) € V X go x V. L’involution o opére dans g par 


o(u,T,v) = (v, -T*, u), 
et, si h désigne l’algebre de Lie de H, 
h={X €g|o(X) = X}={(u,T,u) |we VT € to}, 
ott fg désigne l’algébre de Lie de Ko. Posons 
qg={X €g| o(X) = —X} = {(u, 2L(v), —u) | u,v € V}. 


Sous laction de Ad(H), espace q se décompose en deux sous-espaces irréduc- 
tibles, 
q4 = {(u,2L(u),-u)|ueV}~T.(M) ~ V, 


q- = {(u, -2L(u), -u) |we V} = T_e(M) = V. 
Le cone CC gq défini par 

C = {(u,2L(v), -u) Ju+ve -O, u-ve OD} 
est invariant par Ad(#). Il définit sur M une structure causale invariante et en 
fait un espace symétrique ordonné globalement hyperbolique (voir [2], Theorem _ 


4.1). 
Soit a le sous-espace de Cartan suivant 


a = {(0, L(t),0) | t= tier +---+t,c,} ~ R’. 
Nous choisissons la chambre de Wey] positive 


nS {(0, L(t),0) |O< ty <to<--- < ty}, 
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et prenons Xo.= (0,/,0). Soit +; 1’élément de a* défini par 
yi((0, L(t),0)) =t; (LSisr). 


Alors 
AS ={3 4 -—%) |i< 9}, 


Ar={yll<isr}U{i (i+) li<s}. 
La multiplicité de la racine y; est égale 4 1, et on note d la multiplicité commune 
des racines 4 (7; — 7) et 4 (71 +7;), par suite 


“ don 
=5 2i-—r—-1)-+=—)% =dimV). 
t=1 ( : 7 iy ‘4 af) (n = ) 
(Noter que n = r+ or(r —1).) Les sous-groupes A et No sont contenus dans Go, 
t 
A= {P(exp 9) [t= tie, +---+¢,t,}, 


No est le groupe triangulaire strict (noté N dans [3]), et ANo est le groupe trian- 
gulaire (noté T dans [3]). Le sous-groupe N, est le sous-groupe des translations 
zreeztu (weV). 

Soit 7 élément de G défini par 
r(z) = (e+ z)(e—z) | = -e-2(z—e)}, 
d’inverse 
tz) =(z—e)(zte)-| =e-2z+e)t. 
On montre que H = 70 G(Q) or! (voir [6], théoréme IV.4.3, et [7]). 


LEMME 4.1. Soitw € 2 tel que A(w?—e) £0, et soith =ToP(w)or! EH, 
alors 
h(x) = —a+ P(b)(a—z)7}, 
avec 
a=(wt+w')\(w—w)!, b= Aw—w!)?. 
DEMONSTRATION. Posons y = T~!(z), alors (voir [3], Proposition II.3.3 (ii)) 
A(x) = 7(P(w)y) 
=-e+2(e- P(w)y) 
=—-e+2P(w!)(w? —y)! 
=-e+ Pw ')((x +e)! — (a+e)7") 
En utilisant Videntité de Hua ({3], Chapter II, Ex.5), 


-1 


ub—v! = (ut Plu)(v— ut), 
nous obtenons 


h(x) = —e — P(w')((a+e) + P(a+e)(x—a)“'). 
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D’autre part 
e+ P(w')(at+e) =a, P(w')P(a+e) = P(b). Oo 
Considérons dans M l’ouvert 
NA-mo = {(ge +u, -ge + u) |g © NoA, ue V. 
En posant v = ge cet ouvert s’identifie au tube V x 2. Cc V x V. Nous munissons 
W=V x V de la structure d’algebre de Jordan définie par 
(ui, V1) (U2, vo) = (uu. + V1 V2, UjV2 + V1 U2). 


Nous identifions la sous-algébre {(u,0) | u € V} aValgébre V et posons j = (0, e). 
Alors j? = e, et nous pouvons écrire (u,v) = u+ jv. Cette algebre W est 
isomorphe a l’algébre produit V x V, lisomorphisme étant donné par 


utjue (utv,u—v)=(rz,y)EVXxV. 
Nous noterons 
R(iut+jv) =u, Sut jv) =v. 
LEMME 4.2. Siv est inversible dans V, et u+ ju inversible dans W, alors 


S(ut jv)! = (v- P(uju!)™. 


DEMONSTRATION. Supposons d’abord que u = e, alors 


(e+ ju) ") = (e— ju)(e— 0?) 7! 


? 


et 
S(e + jv)7) = -v(e —v?)7} 


=(v-v7)7}, 
Supposons que u € 2, alors 
ut ju = P(u2)(e+jP(u-2)v), 


et 
1 





S(u + jv)7! =P(u-?)(P(u-?)u — P(u2)v-?)~ 
=(v- P(u)o)™". 
Les deux membres de |’égalité & démontrer sont des fonctions rationnelles qui 
sont égales pour u € 2, elles sont donc égales pour tout u. U 


LEMME 4.3. Soith =r0 P(w)or!, avecw EN, A(e —w?) £0, alors 
$h(jv) = (P(a)o7? — P(B)v)™, 


avec 
a=}(wt+w’), B=}t(w—w""). 


DEMONSTRATION. D’apreés le lemme 4.1, 


h(jv) = —a + P(b)(a — jv), 
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et d’aprés le lemme 4.2, 
SA(jv) = P(b!)(—v + P(a)u7!)™*. 


De plus 
P(b-')P(a) = P(b~1a) = P(§ (w+w")). Oo 
Le lemme précédent permet d’expliciter la représentation intégrale de la fonc- 
tion sphérique ~) ({2], section 5). 


THEOREME 4.4. Posons 
Tr 
p-A= S> SiVis 
i=1 


alors 


avec 


DEMONSTRATION. Puisque 
e(P-A.A(w)) — A;(Sw), 


la représentation intégrale de y, s’écrit 


er(w)= ff a.(9r-w)an= f &,(S(h- w))dh. 


Ko\H 
L’espace Ko \ H s’identifie au céne 2 par lapplication 
h= ko 70 P(w?) or ty, 


et le résultat annoncé se déduit du lemme 4.3. O 
Cette représentation intégrale peut servir a définir les fonctions de Legendre 
généralisées de deuxiéme espéce Q, sur une algebre de Jordan 


Qs(5 (v +v7")) = Es(jv~"), 


pourvé€e4+2. 


5. La fonction cp(A) 


D’aprés le théoréme 6.3 de [2] l’intégrale qui définit y) (A réel) converge si et 
seulement si 


ep(A) = an(71)?**diy <0, 
D 
(notée ca(A) dans [2]) ot D désigne la boule unité de V pour la norme spectrale 


D={xEV||z| <1}. 
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Un élément 7, de N, s’écrit 
M%-z=(z}-u)y! (weV). 


Appliquons 7, 4 je. D’aprés le lemme 4.2 


(m1 - je) = S(je—u)! = (e—u?)71. 
On en déduit que cp(A) peut s’écrire 
ep(A) = A,(e — u”)du, 


fuj<1 


avec 


_ 
A+ p=— So sis %- 
i=1 


THEOREME 5.1. Posons 
r 
= S- NY 
i=1 
alors 


- d d 
y= e]] B05 +5, ee j+de—5th5): 


g<k 


COROLLAIRE 5.2. L’intégrale qui définit ~, est convergente si et seulement 
St 
1 d 
RA; > >) RA; + Ax) > 2 -1. 


DEMONSTRATION. Posons 


J,(s) = As(e — x”) dz. 


jxl<1 


Nous allons calculer J,(s) par récurrence sur r, en adaptant une méthode due 
& Hua (([4], Theorem 2.1.1). Considérons la décomposition de Peirce de V rela- 
tivement a l’idempotent c,, 


V =V(c,,1) + Ver, ) + V (er, 0), 
zg=te,-+€+2% (t€R). 
Soit ¢ la représentation de Vo = V(c,,0) dans EF = V(c,, 5 ) définie par 
P(Zo)E = 2aoé. 
La forme quadratique associée 4 ¢, définie sur F& a valeurs dans Vo, définie par 
(Q(E)|z0) = (P(xo)€I€), 


vaut 
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Notons e, = ¢; +--- +c (1 <k <r). La décomposition de Peirce de e — x? 


s’écrit 
ex? =(1-t? — 5 |léll’)er 
— (b(zo)€ + t€) 
+ (e,-1 — x3 — $ Q(E)). 
En utilisant la relation 


A(x) = Ara(zo)(t— 3 (#(07)EI6)) 
nous obtenons 
A(e~ 2?) = Aya (er-1 ~ 25 ~ 3 Q(€)) P(t), 
ou p(t) est un polynéme du second degre, p(t) = at? + 2bt +c, avec 
a= —1- ((e-1 - 23 - 3. Q(6)) “€l€). 


Soit 7 élément du groupe triangulaire Tp de l’algébre de Jordan Vo associé au 
systéme d’idempotents c),... ,C;—1 tel que 


€r—1 — Le = Tol p-1- 
Il existe une transformation linéaire 7) de l’espace F vérifiant 
Q(7o§) = TQ(E), (7020) = FoH(z0)TG- 
Posons € = V27on, 
er-1 — £4 — § Q(E) = To(er-1 — Q(E)). 
Tenant compte du fait que Q(7) est un élément de rang < 1 nous obtenons 


a=-1- (o(er—1 = Q(n)) n\n) 
IInl|? 


~ 1=|nl? 1 Ini? 
= —A,_i(er-1 — Q(n)). 


On peut montrer que b? — ac = 1. Notons que 





d€ = 23A,_1(ep_1 — 22) 2 dn. 


LEMME 5.3. Soit E un espace euclidien de dimension N. Pour fs > —1, 


ne N 
(1 - ||x||?)*dz = ‘B(s +1, =). 
ee T(¥) 2 
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LEMME 5.4. Soit p(t) = at? + 2bt +c un polynéme du second degré tel que 
a <0, b?-ac=1. Soient a et 8 les racines de p. Pour Rs > —1, 


ie p(t)'dt = B(s + 1,4 )Jal-e-?. 


Nous intégrons d’abord par rapport a t en utilisant le lemme 5.4. Nous 
obtenons 


Fils) = Jpni(8° + )B(sr +1,3)28 / ALY (er—1 — Q(n)) dn, 


ln|]<1 


avec 8° = (s1,..., 8-1), 0) = 8; +5, +1 (1<j<r-1), e 


A®) = Agim A (1<k <r). 
Pour le calcul de lintégrale en 7 nous posons 
m = (Cr-1)n, M =N-M- 
Alors la composante de Q(n) dans Vo(c,-1,0) est égale & Q(mo). Ainsi 
AS~Y (e,-1 — Q(n)) = ASTD, (er—-2 — Q(t0)) Art (€r—1 — Q(m)) 7" 
Puisque Q(7) est un élément de rang < 1, 
A,—1(er—1 — Q(n)) = 1 = Inll?. 


Posons 
28 
m = (1-||noll?)?v, v € O(er-1)E, 
alors 
[a= dol? = tn)? de 
\Inli<1 


= (1= Io?) f 


et, d’aprés le lemme 5.3, 


a — lull?) du, 


vil 


e d d 
= (1 — ||nolI?)°°-*# Blo, + 1, 5)- 


On montre ainsi par récurrence que 
r-1 d d 
/ AS) (e-_1 — Q(n))dn = c ] ] B(oj + (7-5 -1)5 +1,5). 
Iln]<1 jal 2 2 


Finalement, puisque 





d . n 
sj = Apajul i git + 1-2j) = or? 
nous obtenons 
= 11 d d 
=cl] i+3,5)]] AjtAj—=4+1,x). O 
J,(s) ELLER ig) Ler j a7 15) 
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Nous concluons par deux remarques. Dans [2] nous avons déterminé la fonc- 
tion cp dans le cas des espaces symétriques ordonnés de rang un, et dans le 
cas des espaces ordonnés de type Gc/Gr, dits de Olshanski. Dans tous les cas 
connus 





ep(A) =c II (ie — 341,32). 


Poe {a,a) 2 2 
Nous conjecturons que cette formule est vraie dans le cas général. 

Le dual riemannien de l’espace symétrique ordonné M est le tube Tn = G/K, 
ou K = {g € G| g-ie = ie}. Dans [1] nous avons étudié les fonctions sphériques 
de Tm, par des méthodes analogues a celle du présent article. Il serait interessant 
de mieux comprendre la relation qui existe entre l’analyse harmonique de l’espace 
symétrique ordonné M et celle de son dual riemannien Tp. 
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2. Spherical Principal Series for the Lorentz group 


2.1. Notation. We set up some standard material, for more details see for 
example [7,10]. Vectors in R"*! are written as rows with entries indexed by 
0,1,...,n. Equip R™*+! with the bilinear form 

(z,y) = xoyo — (tiyi +.--2nYn), Va,yER™*?. 


The Lorentz group G = SO,(1,n) is the group of (n+ 1) x (n+ 1) real matrices 
g such that (rg,yg) = (x,y), Vz,y € R"*! with det(g) = 1, and with goo > 0. 
We will fix the maximal compact subgroup K to be GM O(n + 1), so that 


K={ € ‘) : ke SO(n) } 


The Cartan involution on G is 0(g) = (g*)~!. We let A be the subgroup 


cosh(t) 0 sinh(t) 
A= a(t) = 0 In-1 0 :teEeR ; 
sinh(t) 0 cosh(t) 


and then have 
:we SO(n—1) >, 


its centralizer in K. The following subgroups are normalized by MA, 





1+$5|c/? « —3|z|? 
N= n(r) = x* | —2* :cER™! 
3lel” e 1—3)/2/? 
and 
1+ 3|z|? x 5|a|? 
V = 4 v(x) = O(n(z)) = —2* Tyo —2* :2é€R™! 
siz? 2 1-5\|2/? 


Note that (1,0, 1)v(x) = (1,0, 1) and (1,0,—1)n(z) = (1,0,—1). We define the 
cone X to be {zc € R"*!: (x,z) =O and zp > 0}. This is an orbit of G since 
(1,0,1)K = {x: a € X and zp = 1} and for all real numbers t¢ 


(1,0,1)a(t) =e'(1,0,1) and (1,0,—1)a(t) =e7*(1,0,—-1). 


The group G has an Iwasawa decomposition ANK and there are functions H : 
G—Rand«:G-— K so that g = a(H(g))n«(g) € ANK for all 9 € G. We 
can find the Iwasawa components of elements of G by considering their action 
on the points (1,0,—1) and (1,0,0) in R"*?. In particular, for z € R™~', we 
have v(x) = a(t)nx(v(z)) in ANK where e~* = e~ 7?) = 1+ |z/? and 


1 0 
a(v(a -( * * 
0 2x/(1+|a|?) (1 |a|?)/(1+ |2|*) 
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2.2. Homogeneous functions. A function f : X — C is said to be homo- 
geneous of degree a if f(tx) = t*f(x) Vx € X,t > 0. Every homogeneous 
function is completely determined by its restriction to 


L={r:2€Xand 2p = 1}. 


The set © can be identified with the unit sphere S"—! in R” by (1,€) € U if and 
only if € ¢ S"—!. If f is homogeneous of degree a and g € G then the function 
xt f(xg) is also homogeneous of degree a. We can also consider this as a 
function of g. If f is homogeneous of degree a on X and if g € G,u € SO(n-1), 
xc €R”! andt ER, then 


(1) f((1,0, -1)m(u)a(t)n(x)g) = ef ((1,0, -1)g). 
That is, the function g +> f((1,0,—1)g) is left 4 AN-covariant. 
Define a function c : S"~! x G — (0,00) by setting c(€,g) to be the Oth 
component of (1,£)g. This follows the device used by Hansen in [9]. 
For € € 5"~! and g € G define €-g to be the element of the sphere S°~! with 
(1,€)9 = c(€,9)(1,€- 9). 


We could write this as 


(1,8) ag = (goo + €910, 901 + €911) 
gio 911 


so that € H €-9 = (goo +€910) (901 + €911) is a fractional linear transformation. 
If f is homogeneous of degree a then 


f((1, €)9) = c(€,9)°f(1,€-9), VEES™ G9 EG. 
Let y denote the normalized rotation invariant measure on S”~!. 
LEMMA 1. If h is a continuous function on the unit sphere S"-! andg €G 
then 
J n@ awe = ff mE adelE.a)' duce). 


COROLLARY 1. If f is a continuous function on X which is homogeneous of 
degree 1 — n then 


| f(1,8) du(é) = | f((1,€)9) dulé), for allg €@. 
Sn-1l Sn-l1 


2.3. Principal Series. If f; is homogeneous of degree a and f2 is homogeneous 
of degree 8 then f, fo is homogeneous of degree a + 8B. Let p = (n — 1)/2. 


DEFINITION 1. For each complex number v let I, denote the space of func- 
tions on the cone X which are: 


(1) measurable; 
(2) homogeneous of degree —p — v; 
(3) are in L?(5"—!) when restricted to the sphere. 
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The action of G by right translation on elements of I, is called a Spherical 
Principal Series Representation, with 


Uv, 9) f(z) = flzg), Ve EX, fel,g eG. 


Furthermore, if € € S"~! then U(v,g)f(1,€) = ce(€,9)-°- "f(1,€- 9). Equation 
(1) shows that we can identify elements of I, with functions on G which have 
the following covariance with respect to MAN, 


F(ma(t)ng) =e?) F(g), Vm e M,tEeR,neN,g eG. 


This identification is made by F(g) = f((1,0,-1)9), Vf € I,g € G. In this 
case, the action of U(v,g) is right translation by g of a function on G. 

If v is purely imaginary and if f, and fo are in I,, then f, fo is homogeneous 
of degree —2p, and 


[Aaonmga =f veana.oow anes a 


That is, the L7($"~') inner-product is invariant under this action of G. 


PROPOSITION 1. Ifv is purely imaginary, then the principal series (1,,, U(v,-)) 
is a unitary representation of G. 


3. Intertwining Operators 


PROPOSITION 2. For each v € C there is a G-invariant Hermitian pairing 
between I, and I_z given by 


(filfe) = des fi (1, €) fo(1, €) dé. 


That 18, (U(v,9)fi|U(-7, 9) f2) = (filfe), Vii € L, fo € Is: 


From now on assume that v is real. 
We will construct an operator A(v) : I, —> I_, with the intertwining prop- 
erty 
A(v) oU(v,g) =U(-v,9) 0 Alv), Vo EG, 
and which is self-adjoint with respect to the L?($”~!) inner-product. 
For such an operator A(v), the pairing 


(fi, fo) > (fil AW) fa) 


defines a G-invariant Hermitian form on I,. 
Consider the function 
(x,y) > (x,y)? 
on X x X. In each variable separately this is homogeneous of degree —p + v. If 
f El, and y € X then 
are f(a) (x,y) Pr 


is homogeneous of degree —2p. 
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DEFINITION 2. For each positive v let A(v) be the operator 


AQ) g(a) = fF #046) (C6).2)- dul) 


It is enough to consider x € U. 


Ae) s(tom) = ff $0.8) (0m) dul 


Question: For which values of vy > 0 is this a positive-definite operator on 
L?($"—1)? Hansen’s paper [9] provides an elementary method for answering 
this question. See also the paper of Faraut [6]. 

The binomial theorem states that 


ig ee vy—p mm _ Va (P~¥)m m 
(1-2) r=¥( a Jen xz "2a L 
LEMMA 2. 
(1) Ifp € L?(S"—1) andm>0 then 


ff. eee mM” dulE)au(n) > 0 
gn-l gn-l 
(2) Ifp € L?(S"-1) and 
[216-9 ante) =0 
for allm>0 andn€ S"—!, then p =0. 


The first part of this lemma is an example from Krein’s theory [16] of positive- 
definite kernels on homogeneous spaces. It follows that if 0 < v < p then A(v) 
is positive-definite. 


PROPOSITION 3. If0 < vy < p= (n—1)/2 then (1,,U(v,-)) can be given 
the structure of a unitary representation of G via positive-definite G-invariant 
Hermitian form 


(fi, fo) > (filA@) fe). 


This is a well known result, for example see [10,15]. 
When v — p is a non-negative integer J, 


ee er ae 3 (2 cure”. 


m=0 ae 


In particular, the operator A(p + J) has finite-dimensional range in I_,_y. 
Each of the spaces I,,, is reducible, since it contains the G-invariant subspace 
ker A(p + J). 
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PROPOSITION 4. If J is a non-negative integer then ker A(p + J) is the sub- 
space 


{fetes f FEE nm)" dule) =0,¥m < ne sr 


PRoorF. For each j > 0 let ;(R”) denote the space of spherical harmonics 
of degree j on R”. Furthermore, let 2;(€,7) denote the reproducing kernel 
for H;(R"). From Krein [16] we know that Z;|gn-1,gn-1 is a positive-definite 
kernel. In fact, it can be expressed in terms of ultraspherical polynomials and 
there are coefficients a,,¢ so that 


Z(En= Yo ajye(E-n)-%, Vénes, 


O<e< [7/2] 


with a; # 0. Rearranging this, we have that 


(€ 5 ny = > Bye Z;-20(€, n); VEN € SPX 


O<e<{y /2] 


with ;,0 # 0. Because the left hand side is a positive-definite kernel on S"~ x 
S"-1 and because the reproducing kernels are the fundamental SO(n — 1)- 
invariant positive-definite kernels, it follows that 


Bye 20, WO<e< [7/2]. 
When we look at the kernel (€,7) + (1-€- ny” we see that it is equal to 


S> cae ZelEsm) — SY ese Ze(E,n) 


o<e<s 1<e<v 
e even «odd 


where all the coefficients cj are positive. For a fixed function f in L*(S"~') 
the functions 


no | f(€)Zs(Esn) aul), 7 > 0, 
Sn-1 


are linearly independent. From this we conclude that f € ker(A(p + J)) if and 
only if f is orthogonal to @J_9H;(R"). O 


4. Derived Intertwining Norms 


We will show that ker A(p+J) carries a G-invariant positive-definite Hermitian 
form. It is enough to consider the action of G and A(v) on functions whose 
restriction to the sphere » are finite linear combinations of spherical harmonics. 
Take two such functions in ker A(p+J), say f; and f.. The intertwining property 
is 


(filAW) fo) = Ug) AlAMU(y, 9) fe) 
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and both sides of this expression are analytic functions of the variable v. Differ- 
entiating with respect to v gives: 


(AIA) fe) = (O'U, NAIAMUY, 9) fa) + UU, NAIA (Y)Uv, 9) f2) 
+ (U(y, 9) filA)U"(, 9) fa): 


Note that this procedure can be included in the setting of Jantzen filtrations, as 
defined by Vogan in Def. 3.7 of [27]. See also, [11], section 2 of [24], and section 
7 of [28]. 


PROPOSITION 5. On the subspace ker A(p + J) CI,4,5 the Hermitian form 


(fi, fo)  (filA"(p + J) fe) 
is G-invariant. 


Go back to our calculation with the binomial coefficients and isolate the terms 
containing (0 — y+ J). Using Pochhammer notation, we have 


(0 -Y)m =(p—v)s(op-ut+JI)\(p-v+J+1)m-s-1 
provided m > J. Letting vy — p+ J we see that 


(p = UV) m J Are 
— —J)s(V)m_—y-1 = (-1)° X positive term. 
oad )uQ)m-s-1 = (-1)" x positi 

THEOREM 1. If J is a non-negative integer then (ker A(p + J),U(p+ J,-)) 
can be equipped with the structure of a unitary representation via 


(fi, fo) > (-1)"(filA’(o + J) fa). 


Johnson and Wallach [12] did the “dual” case, by considering residues of A(v) 
when v is —p— J. 

We can view the space ker A(p+J), equipped with this Hilbert space structure, 
as a space of distributions on S"~! satisfying a moment condition. 

If f; and fo are elements of ker A(p + J) with spherical harmonic expansions 
fc, §) = yas4i Y;(fe,€) on S"~ then there are positive coefficients a(n, J, j) 
such that 


co 


(2) (-1)"(AIA(o+ J) fo) = S2 a(n, J,5) (% (A) ¥5(f2))- 


j=Jt1 


The coefficients a(n, J, 7) can be obtained from calculations in [7, 12]. 

Fabec [4, 5] used the noncompact picture to describe derived intertwining 
norms for SO(1,4) and SU(1,2). Other recent work involving derivatives of 
invariant Hermitian forms has been done by Faraut and Koranyi [8], Arazy [1], 
and @rsted and Zhang [21]. 
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4.1. The non-compact picture. Now take elements of I, to be functions 
on G with left covariance with respect to MAN, as described by equation (1). 
When we rewrite the definition of the intertwining operator it becomes 


A(~)H(a) =f ((1,0,—1)as(1,0,—1)8)°"? $(R) 
Next, restrict g to be an element v(z) of V, so that 
(1,0, -1)v(z) = (14 |z|?, -2z,|z|?7-1), VzeR™?. 


Since all the functions being considered are left-M-invariant, we can transfer the 
integral over A to an integral over V, see (5.25) in [13]. That is, A(v) f(v(z)) is 
equal to 


i (1+ [z[?, -2z, |Z)? — 1), (1,0, -1)«(v(2)))"* (1 + |e?) ~?? f(«(v(2))) do. 
But f(«(v(x))) = (1+ |z|?)°*” f(v(x)). Hence, we have that 


(3) Av) f(v(z)) = 2°? | Jc — 22-2 f(v(n)) de. 


Rr-l 


In the following material we will be switching back and forth between functions 
on G and functions on R”~! and so we use some notation to distinguish the two 
settings. Take an element f of I,, restrict it to the group V, and then identify 
it with f? on R™~! by assigning 


f(z) = f(v(z)), Vee R™™. 


For each v € C we translate the action of principal series restricted to MAV to 
an action on functions on R”~! by 


(U(v, m(u)a(t)v(2)) f)’ (2) = f(v(z)m(u)a(t)v(2)) = el) fP(e'zu + 2), 
for all u€ SO(n— 1), t € R, z,z € RR" and f €I,. Our hypothesis is that 
f\x isin L?(M\K) and (5.25) in [13] says that this means 


J lstutayy Pete de < 0 
R.-1 


But f?(z) = e~ t+»)(A(v(@)) f(«(v(z))) and v is real, so this inequality can be 
rewritten as 


(4) Es, [f((v(2))) Pe 2H) dz = , ; |f?(x)|?(1 + ||?) ees 


This inequality can also be used to show that elements of I, are integrable on 
R"~' when v is sufficiently large. 
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LEMMA 3. Suppose that J is a non-negative integer. For all F € 1,45 and 
every polynomial p on R"—" of degree less than p+2J the function x ++ F(zx)p(z) 
is integrable on R"7!. 


PROOF. It suffices to treat a polynomial p(x) which is homogeneous of degree 


m. Then 
Pe .. 


and this is bounded by a constant multiple of 


(/..., [F*(a)|?(1 + |2|?)?e+ ic) (Meare 


The final integral will be finite provided m < p+2J. O 

In the case when v = 0 we can identify Ig with L?(R"~') and principal series 
is unitary. In fact, U(0,a(t))f(v(z)) = e?* f?(etx) is the usual dilation of an 
element of L?(R"~'). A Fourier transform argument can be used to show that 
Ig is irreducible under the action of U(0,-)|;7;av provided n > 2, see Theorem 6 
on page 151 of [20}. 

The kernel of the intertwining operator A(p+ J) is characterized by inequality 
(4) above and a moment condition. 





p(a\F*(2)| dz < ep f (1+ lel?) [F(2)| dz 
Rz7-1 


PROPOSITION 6. A function f in I,,5 belongs to the kernel of A(p+ J) if 
and only if 


i P,(z)lz(*f(2)dze =0, VO<s+k< JP, €H,(R"). 
R.-1 


PrRooF. We use Proposition 4 and the formula 
| F((1,0, -1)k) dk = | F((1,0, —1)x(v(x)))(1 + |2|?)~?? dx 
K Vv 


to say that f € 1,,,) is an element of ker(A(p + J)) if and only if 
(6) f Ple)(@e-m-+ (1 — [alma + le de =o, 


for allO <m< J, € R"! and no € R. Suppose that f € ker(A(p + J)) and 
treat the previous integral as a polynomial in 72, then 


(6) Ts f(z) (2 : m)i(1 = |x|?) (1 4 |x|2)7-™ dz = 0, 


for allO < 7 < m < J and m € R”!. But ker(A(p + J)) is closed under 
dilations, so that dilating f? in (6) and changing variables implies that for all 
A> 0 


ie f?(a)(a-m)? A? = |x|?) 9 (0? + |z|?)2- dz = 0, 
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for alO<j<m< J,m €R”!. Take the case m = J and treat this as an 
even polynomial in the real variable A. We see from the binomial expansion that 
[ f'(x)(@-m)'|2)?*de=0, VO<jtk<JmeR™!. 
R--1l 
This implies that 


/ f?(x)Z;(z,m)|z/** dx =0, VO<jt+k<J,meR™, 
R-1l 


where Z, is the reproducing kernel for the space of spherical harmonics of de- 
gree 7 on R”~!. This implies the formula in the statement of the proposition. 
Conversely, this collection of equations implies (5). O 

This tells us that the elements of ker(A(p+J)), viewed as functions on R™~?/, 
have zero moments up to order J. The following generalization of Pitt’s inequal- 
ity comes from the proof of Theorem 2 in the paper of Sadosky and Wheeden 
[23]. 


PROPOSITION 7. If J is a nonnegative integer and f € ker (A(p+ J)) then 


i IPL) ae 
weil RP} 


1/2 
sens (fi LMe)PO+ leva) © <oo, 


1/2 





Proor. In proving theorem 2, on pages 532-3 of [23], Sadosky and Wheeden 
require the following properties of the function f and the weight w: 


(1) that f(x)p(z) be integrable on R” for all polynomials p of degree < k; 

(2) that f,, f(x)p(x) dx = 0 for all polynomiais p of degree < k. 

(3) that the weight w be one for which the inequalities (8) on page 523 are 
valid. 

Take w(t) = 1in Theorem 2 of [23] and set p = q = 2, so that @ = 2(n—1)—n= 
n—2. The weight w there is w(t) = |t|"~!. In this case it follows from the 
methods used in the proof of the theorem that if f satisfies conditions (1), (2), 
and (3) above, then 


(ee “) © seua(fePis* an) 


~~ NS 


1/2 
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If we also assume that f € L?(R”) then we can say that 


| Rel a \"” 
ro EY Tel 
jle\’ a \" Peon 
< (fa. (2) é) a ie (Be &) 
“A 2 1/2 
ROLY lide — 
(88) 8)" (Lnere 


1/2 
sena (fh Ise)lai ae) + I 


Notice that if 2k—1+n > 0 then |x|?*-1+" = |a|?*#-24"41 < (14 |a|?)F-U+@ 4/2, 
for all z € R”. Since n > 2 implies (n + 1)/2 < n, the previous inequality can 
then be written as 


ox 2 1/2 
ely a \ . ited aesetcan 
a (/.. (Be) «| <ena( fish + lyteae) 


In the case of f € ker(A(p+J)), use Lemma 3 and Proposition 6 to check 
conditions (1) and (2) above. We conclude that if J = k — 1 and we are dealing 
with functions on R”~! instead of R”, then f € ker(A(p+J)) implies the 
inequality in the statement of the proposition. UO 

Pitt’s inequality and the moment condition (Proposition 6) shows that there 
is a MAV-equivariant mapping 


P; : ker(A(p + J)) — L?(R"7') © Ip 























given by 
(8) (PyF)(€) = lel? 7 F(6). 
It is straightforward to see that P, satisfies 
P;oU(pt+J,g) =U(0, 9) o Py, Vg € MAV. 


From its definition, the operator P; has trivial kernel. We apply Schur’s Lemma 
to see that P, is a constant multiple of an isometric isomorphism between the 
kernel of A(p+ J), equipped with the derived intertwining norm, and L?(R"~!), 
equipped with the usual norm. 


THEOREM 2. Suppose that n > 2. For each nonnegative integer J there is a 
positive constant Cn,z such that f € ker(A(p+ J)) implies that 


(-1)" (A'(o+ J)f| f) = cus f Fol \e|-24 2° ae. 
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4.2. Comments about radial functions. Suppose we start with a function 
F on K which is bi-M-invariant, extend it to be an element of I,;,, and then 
restrict it to V. The function on R”~! obtained in this manner is 


F(z) = eM HV) F«(v(2))), Vee R™. 


If F is both left and right M-invariant then k +> Fk) is a function of the entry 
knn- But «(v(2))ajn = (1 — |z|?)/(1 + |z]?). 


LEMMA 4. An element F of I, has F|x bi-M-invariant on K if and only if 
F° is a radial function on R"—!. 


Now assume that F € ker(A(p + J)) and that F° is radial. For every k > 0, 
the function x +> |z|?* F?(z) is radial. Lemma 3 says that if P, € H,(R”~) with 
s>Oands+2k <p+2J then z+ P,(x)|z|?* F’(z) is integrable. In addition 


is P,(x)|x|?* F’ (x) dx = 0, 
Rr-1 


since we can change to polar coordinates and use the fact that spherical har- 
monics of degree greater than zero are orthogonal to constants when integrated 
over the unit sphere. Every polynomial on R”~! can be written as a sum of 
homogeneous polynomials. Every homogeneous polynomial of degree m can be 
written as a sum of polynomials of the form P,(z)|z|?* with s+ 2k =m and P, 
a spherical harmonic of degree s. 


LEMMA 5. If F € ker(A(p+ J)) has F° radial then 


/ p(z)F"(x) dx = 0 
R-1 
for all polynomials p of degree < 2J. 


PROOF. We can assume that p is homogeneous of degree m < 2J. We can 
then assume that p(x) = |z|?*P,(xz) form = 2k+s and P, € H,(R"~'). Ifs >0 
then the argument above shows that p is orthogonal to F’. If s = 0 we can use 
Proposition 6 and the fact that 2k < 2J to complete the proof. O 

Using inequality (7) in the proof of Proposition 7 for a function with zero 
moments up to degree 2J, we see that 


~ 2 
(9) ee (a | dy < en ie |F?(z)|2(1 + |x|?)27+20 dx is 








lyl27 lyjP—t = 
Next, apply the operator P,;, defined in (8). 


COROLLARY 2. If J is a integer with J > p and if F € ker(A(p+J)) has F° 
radial then 


—— 2 
Res (Far) oes < ae [F*(2)|2(1 + |2|2)22+2¢ a 
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2 
| po 2It2e-) dp oo, 


0° Se 
In fact, if € is a unit vector in R™! then [ [Ps FP(ré) 
0 
and the value of this integral 1s independent of the choice of €. 


This is the condition on P;F° required for Caldéron’s formula to apply, see 
section 3.12 of [19]. 


5. Alternative Realizations 


In this section we will show that the representations (U(p+J), ker(A(p+J))) 
can be realized in the kernel of a first order invariant differential operator acting 
on 7-covariant functions on G. This is similar to the realization of discrete 
series found by Knapp and Wallach [14]. We also show that although they are 
not discrete series representations, they exhibit some square-integrability when 
restricted to MAV. 


5.1. First order differential operators. There is a standard construction 
of a gradient operator V. The complexified Lie algebra gc is identified with 
right-invariant complex vector fields on G. Let po be the —1 eigenspace of 6 
in gc. For each unitary representation (7,H,) of K, let C°(G,7) denote the 
smooth left 7-covariant functions on G. There is the canonical gradient operator 
V defined by 


VF(9)(X)=Xf(9), VEE C™(G,7),9 € GX € po. 


The value V f(g) is an element of Homc(pc, H,), which is identified with H, ® 
po. Let (Ad*, pg) denote the co-adjoint action of K. Then 


V :C™(G,r) — C™(G,r @ Aad*) 


is G-equivariant. Every K-equivariant projection E on H, @ p* gives rise to a 
G-invariant differential operator Eo V acting on C(G,7), see [3, 14]. 

In all that follows we fix an integer J > 0 and let (7741,H +41) be the action 
of K on the spherical harmonics H ;,:(R”). Since po = H,(R"), the ten- 
sor product 71)41(R”) @ pg includes the K-invariant subspace Hj+2(R”) with 
multiplicity one. Let Fy be the projection on to the orthogonal complement 
of Hy42(R") in Hy4i1(R”) @ pE. Following [2], the differential operator 6, is 
defined to be 

Oy = By oV. 
5.2. Cauchy-Szegé maps. For fixed J > 0, as in the previous section, let 
Ry; : © — Hy j4i1(R”) be an identification of the complex line with the one- 
dimensional subspace of M-fixed vectors in 1 ;4;(R”). Take @; to be the M- 
fixed unit vector in H)41(R”) given by R (1). We use this to construct Cauchy- 
Szeg6 mappings, 


Syv : I, —+C(G,7341), where Sy,f(g) =f T341(k)~' Ry f (kg) dk. 
K 
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Such an operator Sj, is an example of a 7-quotient map, as defined in [17]. For 
recent work on general r-quotient maps for all real-rank one semisimple groups 
see [26]. The key property is that they do not annihilate the K-type 7;41, see 
lemma 3.5.1 in [18]. 


PROPOSITION 8. For every v € C the image of Sj, is non-trivial and it 
contains the K-type (7341, Hy41) with multiplicity > 1. 


Let Nz, be the image under S,, of the G-invariant subspace of I,, generated 
by the K-type (7741, H +41). For the following result, see [11, 17]. 


PROPOSITION 9. (Kunze) For everyv € C the subspaces Ny, and Nj, are 
identical in C(G, 7341) 


Recall from Proposition 4 that the K-type 7,41 occurs in ker(A(p+ J)) and 
that ker(A(p + J)) is irreducible under the action of G. This means that the 
ker(A(p + J)) has a nontrivial image under Sj, and that it is equal to the 
image of N;-,-,. Arguing as in section 5 of [18], the image of S;_,-y is 
contained in the kernel of the invariant first-order differential operator 6) acting 
on C™(G, 7341). 


PROPOSITION 10. For J > 0 the Cauchy-Szegé map Sjp45 takes ker(A(p + 
J)) into the kernel of the differential operator 6; and its image is an irreducible 
subspace which includes the K-type 7 +1. 


5.3. Main Result. From Johnson and Wallach [12] we know that these rep- 
resentations of G are not square-integrable. However, we claim that for J large 
enough the image Vj,p4, is contained in the subspace 


u EO (Gitjei) 2 io \(f(mav)|¢5)|? d(mav) < coh. 


We will use our earlier discussion of both the compact and noncompact pictures 
of principal series to prove this claim. 

The adjoint of the projection Ry : C — Hy41 is R* : Hy41 — C with 
R*+w = (|). When applied to elements of the range of Sj, this is 


R5Ssvf(o)= | (alrrsalb oa) (bo) dk = iE U(v, 9) (k) (era (B)ba1da) ak. 


Let us put ®7(k) = (7741(k)¢y|¢,) for all k € K. In the L?(K) inner product 
we have 

R5Savf(9) = (UY, 9) fF | Ps). 
Fix vy = p+ ZJ and extend ®, to be an element of.I,,,. The function @; is an 
element of ker(A(p + J)), it belongs to the K-type 7741 in that space, and our 
earlier discussion shows that there is a nonzero constant a, such that 


(~1)/(A'(p + J)Os|F) =05(O;|F), VF € ker(A(p + J)). 
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This means that 
R5Ss prs f(g) = az" (U(p t+ J, 9) f |A'(o + J) Os), VF € ker(A(p + J)),9 € G. 


We are interested in the behaviour of R5S;,1;f on MAV. The function ®, is 
left and right M-invariant on K and so the function x H ®;(«(v(a))) is radial 
on V. 

In our calculations with the operator P; we saw that if f; and f are in the 
kernel of A(p + J) then 


(-1)) 404 Dhl fa) =e f Pufh@PiR@) az 
V 
Furthermore, P,; intertwines U(p+J,g) and U(0, g) for all g € MAV. From this 
it follows that f € ker(A(p+ J)) implies 
R3S yj 4sf(mav) = cf (v0o, mav)P; f”) (x) P ®', (x) dz. 
V 


The image of R*S 3.4, consists of matrix coefficients of the unitary represen- 
tation U(0,-)|aav acting on L?(V). The right Haar measure on M AV is given 
by 


i F(mav) d(mav) = ol Vs F(ma(t)v(x))e~ 7?! dx dt dm. 


Let Q be the function P;®’, on R™-!. 
In the equation above, U(0, m(u)a(t)v(z)) F(z) = e? F(e'zu+ z), and 


‘i et F(e'am+z)Q(x) dz = a F(y+z)Q(e*y) dy =e F«Q,(z), 
R.-1 Rr-l 


with Q;(x) = Q(e~*x). The Fourier transform of this is Q;(€) = e2°Q(eté). The 
square of the L?(R"—! x R, e~ 2? dzdt) norm of e~?* F * Q;(z) is 


| et x Q,|[Rat = / | 
R RJR*-1 


Recall that Q is radial. For a given € # 0, let e” = |€|, so that 


[wes «- [Jo(erg)|s 


But integration over the line is translation invariant and so we need only estimate 


a 2 
te |O¢e*e)| dt when € is a unit vector. Now take |€| = 1 and let s = e’, 


(10) [ \acof a= [" ace)’ S. 


This last expression is independent of the choice of unit vector €. 
In the discussion about moments of radial elements of ker(A(p + J)) Pitt’s 
inequality showed that the integral (10) is finite whenever J > p. 


PO) \Oee| deat 
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THEOREM 3. (Main Result) If J is an integer with J > p = (n~1)/2 and 
n> 2, then the Cauchy-Szegd mapping S44 takes ker(A(p+J)) into the space 
of T341-equivariant functions F on G with the properties that 


/ |R*F(mav)|? d(mav) < oo and 3;F =0. 
MAV 
In fact, there is a constant c > 0 with 
i |R*Syp4sF(mav)|? d(mav) = c(—-1)7 (A'(p + JF IS), 
MAV 


for all f € ker(A(p+ J)). 


PROOF. We have done almost all of this already. The last identity is another 
argument based on Schur’s lemma, since the L?(MAV) norm is a right-M AV- 
invariant Hermitian form on a space which we have already seen to be irreducible 
under MAV. O 

Notice that this says that f + ||R>Sy4sf\|b2Qwav) is a G-invariant norm 

on the irreducible space ker(A(p + J)). 
5.4. Affine symmetric spaces. Schlichtkrull [25] has shown that there is a 
G-equivariant mapping from ker(A(p + J)) to an L?-eigenspace of the Laplace 
operator on the non-Riemannian symmetric space SO,(1,n — 1)\SO,.(1,n). See 
also [7, 22]. 
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THE HOLOMORPHIC CAUCHY-SZEGO 
KERNEL FOR NONHOLOMORPHIC DISCRETE 
SERIES REPRESENTATIONS OF SU(2,1) 
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Dedicated to Ray A. Kunze 


ABSTRACT. We consider some problems of integral geometry on the manifold of 
flags in CP” and discuss their connections with complex analysis and the theory of 
representations. 


Iam happy to be among friends and colleagues of Ray Kunze who can pay tribute 
to this remarkable mathematician. I believe that Ray and I are from the generation 
in the representation theory for whom explicit formulas have the highest priority. 
It is the reason why I chose to talk about some very simple and explicit formulas 
in the conference in his honor. 

I will discuss the analogue of Cauchy-Szeg6é operator for discrete series — the 
intertwining operator from nonunitary principal series into a discrete series rep- 
resentation. This fundamental construction is due to Knapp and Wallach [1, 2]. 
For SL(2,R) the kernel function is just the Cauchy kernel for derivatives of holo- 
morphic functions. My aim is to relate this operator (in the multidimensional 
case) with complex (moreover, holomorphic) analysis. It is another step in the 
general program of holomorphic realizations of nonholomorphic discrete series [3]. 
On the other hand, the classical Cauchy kernel should not be tied too closely with 
SL(2,R): its tie lies rather with the complex line (affine or projective), the con- 
struction should make sense for any domain in there and it is only for the special 
case of the circle that the group interpretation does arise. We want to understand 
which structures define the Cauchy-Szeg6é kernel, with an eye on generalization to 
other groups. It turns out that complex groups are the natural setting that provides 
the essential structures (indeed, more generally, non-group structures also give the 
natural setting). 
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1. GEOMETRICAL NOTATIONS 


We will consider the simplest example - the group SU(2,1). This group acts on 
the projective plane CP? with the homogeneous coordinates z = (zo, 21,22) and 
preserves the domains 


(1) Ds = {z € CP? : |ag|? — |21|? — |zol? 2 O}. 


Let us emphasize that practically all our constructions will deal with the projective 
geometry of CP”, rather the domains (1). (In other words, the group SL(3,C), 
rather than SU(2, 1), is involved.) More specifically, instead of the domain D, in 
(1), we can consider any convex domain D, Cc CP? with (for simplicity’s sake) 
smooth boundary. 

Let CP? = (CP?)’ be the dual projective plane 


(2) (Z, 2) = Z'z;, = Z°29 + Zz, + Zz. 


Here, we identify Z € CP with the line defined by (Z,-) = 0 in CP?. Let (D’)4 ¢ 
CP3 be the set of lines Z which are contained in D_; then (D’), will be the convex 
domain in CPZ (dual to D_), whereas (D’)_ will correspond to the lines in CP? 
which intersect D,. 

Let F be the manifold of flags in CP?: 


(3) F = {(z,Z),z€CP?,Z € CP3,(Z,z)=0}, dim F=3, 
and D the domain of those flags (z, Z) with z € D_ and the line Z intersects Dj: 
(4) D={(z,Z) € F,z€ D_,Z€ (D’)_}. 


If B is the (common) boundary of the domains Dx, let B be the set of flags such 
that z € B and Z is the tangent line to B at the point z. Hence, B is a (real) 
three-dimensional manifold in F' which is a part of the boundary of the domain D. 
It is totally real and, in a certain sense, the Shilov boundary for D. The manifold 
B is isomorphic as a real manifold to the boundary B, but B has CR structure and 
the map B — B destroys this structure (B is totally real but a memory about CR 
structure on B “lives” in the way that B is embedded in F’). 


2. THE CAUCHY-SZEGO KERNEL (FIRST VERSION) 


Nonunitary principal series are realized in the sections of line bundles on B. They 
can be thought of as homogeneous functions satisfying 


(5) f(Az,uZ) = Aw? f(z,Z),  f € P((a, 8). 


We will consider only the simplest case when a = @ = —1 and will denote this 
space of (smooth) section as S(B). The corresponding kernel is analogous to the 
classical Cauchy kernel for functions rather than for derivatives. 

Let us define the standard “holomorphic” volume element on F: 


z,dz, dz] A[U, Z,dZ] 


(U, 2) S Hy SO AU yO. 





(6) 2,2) =! 
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Observe that in the determinant 
[@1, @2, a3] = det(a,, ae, a3) 


if some columns a, are forms we use the exterior product for the computation of the 
determinant. The form 2 is quasi-invariant under the projective group SL(3,C) 
and independent of U when 2 is restricted to F (but not on the CP? x CP3). 
Sometimes, it is convenient to define the line Z which goes through z by introducing 
another point v on the line. Then 


Q(z, v) = [z, dz, dz] A [z,v, dv]. 


Remark. The formula (6) looks asymmetric between z and Z. But the origin of 
(6) is explained by the formula 


7 [z, dz, dz] \|Z, dZ, dZ| 
pai (Z, z) 





and the asymmetry and usage of an element U from which 1) is independent are 
the result of the computation of the residue on F’; we can interchange Z, z and will 
obtain the same form on F’. More precisely, we will obtain the form which will be 
different on CP? x C2, but will coincide with (6) when restricted to F. 


Now we are ready to describe the first version of the Cauchy-Szegé operator: 
a ae on ; 


This operator, acting on the sections f € S(B), yields sections of the holomorphic 
line bundles 3(—1,1) on D, x (D’);. Note that f depends on 3 variables and Sf 
on 4 variables. So we expect that the image must satisfy an additional condition. 
Indeed, we have 


O(Sf) = 0, 


(8) peas Ot a OE 
~~ Ow,OW* ~ AupOW® Ow, OW! Ow. 0W?" 








Remark. Let us remark for comparison that the classical Cauchy formula in pro- 
jective form (on CP!) is the operator on sections of the line bundle #(—1) with the 
kernel 

[z, dz] 


Zs Zz)” = (20,21), Z = (2:2) 





We will denote the space of (holomorphic) sections of line bundle #(—1,—1) on 
Di x (D’) + satisfying the equation Og = 0 as A(Di x (D’),). So S is the operator 
from S(B) into A(Dz x (D’)+). 


To prove (8) it is enough to differentiate directly (7) and to use the condition 
(z,Z) = 0. 


78 SIMON G. GINDIKIN 


Let us emphasize that the construction of (7) depends only on the projec- 
tive geometry and makes sense for any convex domain D,. Also, we can re- 
alize Dy. x (D’)4 as the domain in the homogeneous complex (Stein) manifold 
M = SL(3,C)/GL(2,C). 

We can say a little bit more about the homogeneous domain D, in (1). The 
operator S is an intertwining operator and we have the representation of SU(2, 1) 
on the space A(D4 x (D')4). 

Inside D, x (D’), there is the real submanifold 


(9) A= {(z, Z) 12> (20, 21, 22),Z = (2, ~2Z1, —22),z € Di}. 


The restriction of the operator 0 on A will be elliptic and all solutions of Ug = 0 on 
A admit holomorphic extensions on D, x (D'),. The operator D| , is the analogue 
of the Schmid operator for this case and if we restrict (7) to those (w,W) € A we 
will obtain the Szegé operator of Knapp-Wallach [2]. 


3. THE GEOMETRY OF THE OPERATOR 1 


The operator S sends any object subsisting on B (which is the essential part of 
the boundary of the domain D C F) to an object on the domain Dy x (D’). Cc M. 
What is the relationship between D, x (D’), and D? We can think of (w,W) € 
D4 x (D’)4 as rational curves E,, w in the domain D: 


(10) Eww ={(z,Z) € F: (Z,w) = (z,W) =0}, Eww CP’. 


Using this interpretation we can give a geometrical interpretation of the operator 
QO. For each curve E,,,w, consider the isotropic cone V,,,y C M formed by those 
(z,Z) € M so that the curves EF, w and E,,z are intersecting. Let V...w be the 
infinitesimal image of this cone in the tangent space T,, w.M. We obtain a (complex) 
conformal structure on M > Di x (D’)4 (a version of Pliicker construction). Then 
OD is just the Laplacian of this structure. The restriction of this conformal structure 
on the real submanifold A Cc M will have the Riemann signature. It corresponds 
to the fact that the corresponding curves EF, w,(w,W) € A, are the fibers of 
the (real) fibering of D Cc F over Dj (i.e., these curves do not intersect). Of 
course, the existence of such a fibering (an analogue of Hopf fibering) holds only 
for homogeneous domain (1). 

On M we will use coordinates different from the Stiefel coordinates (w,W), 
namely, we will pin down (w,W) by a pair of flags (u,U),(v,V) from the curve 
Ew,w with the extra condition 


(11) (w,U) = (w, V) = (W,u) = (W,v) = (u, U) = (v, V) = (u, V) + (v,U) = 0 
Then the operator (7) has the form 


F(z, 2) QC, Z) 


PIOUS eal a) 


We will use (w, W), (u,U), (v, V) simultaneously in various combinations. 
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Remark. Of course, we can represent the components of w(W) as the minors of 
the matrices (U,V) and (u,v) respectively but the simultaneous use of all these 
coordinates makes the formulas more compact. 

For example we can rewrite the operator UO as 


] e) 0 0 


(12) 0’ = (wa (Wap) = (wa )(W ar) 





a _ a a a . . . 
where wa, = wo Buy + Vida; + W2guq> ete. Using the identity 


rs) re) 
“ou OW 
we can also write U as 
Pe See ee 6) 
"= Vag llugz) — gs) U ag) 


These forms may seem more complicated than (7) but they uncover a structure 
which is important for applications. 


4. HOLOMORPHIC COHOMOLOGY AND THE SECOND 
VERSION OF THE CAUCHY-SZEGO FORMULA 


The operator S relates the boundary of one complex domain PD to another com- 
plex domain D, x (D’),. This situation is not quite usual for boundary value 
problems in complex analysis. We would like to have constructions more closely 
related to D. The problem is that the domain D is not a Stein manifold (the domain 
D4 x (D’)x is) and has so far no holomorphic objects there. This is the reason 
why we need to consider nonholomorphic discrete series. 

Let us give to the formula (7) the following interpretation. Let D be a manifold of 
those pairs of flags {(u,U),(v,V)} from D so that there is a curve FE, w,(w,W) € 
D4 x (D’)4, which contain both of the flags. In other words, it means that the 
point w is the intersection of the lines U and V (and must lie in D,) and the line 
W goes through the points u,v (and must lie inside D_, ie, W € (D’),). The 
manifold D is already a Stein manifold and we have the canonical covering 


(13) n:D—D,n((u,U),(v,V)) = (u,U). 


The manifold D is 1-pseudoconcave and instead of holomorphic functions we can 

consider a 1-dimensional 0-cohomology there (it is the form in which nonholomor- 

phic discrete series are realized). Due to the existence of the Stein covering (13), 

we can follow [3] and describe such a cohomology in the holomorphic language. 
Namely, if g € A(D; x (D’)+) let us consider the 1-form 


(18) 59 = apy { (wg )a- Wade) + (W555 )a- (w, AV) + 9-((V, au) — (0, €¥))} 


(where we mix coordinates u,U,v,V,w,W). We have holomorphic forms on the 
manifold D with differentials along the fibers of 7: D — D only. 
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The equation Ug = 0 is equivalent to the condition that the form Kg is closed 
along the fibers of 1: it is closed as a form on (v,V) for any fired (u,U) € D. So 
we have closed holomorphic 1-forms with parameters ¢(v,U | v,V,dv,dV). The 
forms «xg satisfy an extra condition: 

(H) ¢, as a function from the parameters (u,U), is constant along the curve 

Eww connecting (u,U),(v,V): d= o(w, W; dv, dV). 

We can consider the cohomology for such complexes of holomorphic forms and 
the condition (H) picks up then the unique representative in each cohomology class 
(holomorphic Hodge theorem {3]). 

We will denote the space of holomorphic forms ¢(w,W;dv,dV) satisfying the 
condition (H) as C(D). In this way, in the homogeneous case we have the realiza- 
tion of the representation of SU(2,1) as the space L(D) and « is the intertwining 
operator from A(D, x (D’),) to £L(D). 

The composition « o S gives us the second version of Cauchy-Szeg6 operator 
which sends sections on B to forms on L(D): 

(15) 

(0S) f(w, W; du, dV) 

1 f(z, Z) Q(z, Z) 


= Ta W Shy Gar ZW, 2)? {(V, z)(w, Z)(W, dv) + (uv, Z)(W, z)(w, dV) 





+ (w, Z)({V, du) ~ (v,dV))}, 
(w,W) € Dy x (D')+,(W,v) = (V,w) = 0. There is a very simple operator from 


L(D) in the Dolbeault cohomology H@)(D) = H\)(D,9(-1,~1)). Let T be a 
section (of course, nonholomorphic) of the fibering 7: D — D, (v,V) = y(u,U). 


Let us consider the operator 
R: L(D) — HD), 
PE L(D) > {GT}OY 
where we push down ¢ to D via the pullback through TI and then we take the (0, 1)- 
part of the form obtained. This operator gives a 0-closed (0, 1)-form on D and we 
descend to the cohomology level, we will obtain an operator R : L(D) + H“)(D) 
which is independent from the choice of the section. This gives an isomorphism 
between these spaces. The composition of the operators (15) and (16) gives a new 
version of the Cauchy-Szeg6 operator: 

RonoS:8(B) > H"(D). 


Remark. In the homogeneous case (1) it is convenient to use the fibration A of D 
over D, to get an explicit construction of the section [. 


(16) 


Lastly, we will construct the intertwining operators which maps the Dolbeault 
cohomology to the solution space of Dg = 0. This is accomplished through the 
Penrose operator P, which is essentially just the integration of the 0-cohomology 
in D over curves E,, w. If Y is a 0-closed (0,1)-form on D with coefficients in 
0(~1, ~1), then 


(17) PY (w, W) = [ Me si! a On cml dry + Tl dro), 
CPi 
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where (w,W) € Di x (D’)3, (u,v) EW, w = UNV, (w,V) = (v,V) = (u, VY) - 
(v,U) = 0. It is easy to check directly that PW € A(D + x (D’);) and that 
O(PY) = 0. As a result, we have the following diagram relating four spaces 


: A(D+ x (D’)+) + £(D) 
7 R 
(18) 4 
S(B) P 

HY (D) 
This diagram is commutative up to a choice of the normalized factors. 


5. INTEGRAL GEOMETRY ON F 


The Penrose transform P (17) is related to the problem of integral geometry 
for 0-cohomology in the domain D C F. Now we will consider the problem of 
integral geometry on the whole manifold F. We will consider the space S(F’) of 
homogeneous functions 


(19) f(Az,nZ)=A“"w"f(2Z),  (z,Z) = 0, 


which are C® (therefore, real functions) and we will integrate them along rational 
curves E w, (w,W) € M. In other words, 


(20) 
flu, Ulu, V) = : f(tow+71v,70U + 71V)(~—T0d71 + T1d70) A (~71d71 + 71 d70) 
CP} 


= [ flz, Z)6([z, w, v])4([Z,U, VI)A(z, Z) A We, D), 


where (u,U) = (v,U) = (u,V) + (v, V) = 0. The problem of integral geometry is 
to reconstruct f from f . It is a over-determined situation and we will describe the 
solution using the «-trick of Gelfand-Graev-Shapiro. Let « be the operator (14) 
and & be the complex conjugate operator. Then for arbitrary (u,U) € F the form 
(k \&)f is the closed (1,1)-form on (v,V) € Fy for the arbitrary (v,V) € F (we 
require u # Av,U # pV). It is commonplace to expect the following (in fact, true) 
relationship 


(21) / (eAR fae) f(uU), (wv) EF 


for any (real) two-dimensional cycle y € Fy,y where c(y) depends only from the 
homology class of 7 and is not zero for the generic case. It is simple to compute 
c(y). Let us observe that formula (21) for V = const is the inversion formula for 
the Radon transform on CP?. 
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Remarks. 1. The cases A(a, 3) for other a, 3 can be handled similarly, but the 
formulas are more complicated. We will obtain in this way all unitary representa- 
tions of non-holomorphic discrete series for SU(2,1). The most important point is 
that instead of the equation Og = 0 we will have an associated system of first order 
equations (zero rest mass equations for suitable spinor fields). 

2. Our construction is related to the homogeneous (with respect to SU(2, 1)) 
domain D Cc F’. There are two other homogeneous domains in F: flags (z, Z) with 
z € D, or with Z € (D’),. These domains fiber over D,,(D’)4 correspondingly, 
in both cases the fibers are projective lines. They correspond to the holomor- 
phic and antiholomorphic series representations. In the formulas for Cauchy-Szegé 
kernels (cf. (7)) we will have powers of (w,Z) only or (z,W) only. Hence, the 
integral formula will yield holomorphic functions in D, or the Penrose transform 
1-dimensional cohomology in D_ (holomorphic functions in (D’)+). 
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Invariant Theory for Boundary Components 


KENNETH I. GROSS 


ABSTRACT. This paper, the first in a series devoted to vector-valued har- 
monic analysis on the boundary of a symmetric cone, concerns finite- 
dimensional representation theory associated to the boundary components 
of the cone of positive-definite n x n Hermitian matrices. The invariant- 
theoretic results in this paper are crucial to developing a theory of operator- 
valued Bessel functions on boundary components, which in turn leads to 
new realizations of singular unitarizable highest weight representations and 
to connections with the oscillator representation. 


1. Introduction 


This paper begins a study of vector-valued analysis on the boundary com- 
ponents of a symmetric cone. Here, we develop the finite-dimensional represen- 
tation theory and invariant theory associated to the boundary components of 
the cone 2 in C"*” of all positive-definite Hermitian matrices. The results in 
this paper, especially the detailed study of the singular operators on the bound- 
ary (Section 5) and the unitary equivalence of Hilbert spaces on the boundary 
(Section 6) are of independent interest. But this paper is ancillary in nature, 
preparing the way for later work that will relate operator-valued Bessel func- 
tions for boundary components to oscillator representations and to new analytic 
realizations of the most singular unitarizable highest weight representations for 
the automorphism groups of Hermitian symmetric spaces. These latter repre- 
sentations may be said to lie in the vector-valued Wallach set (cf., [6]) for these 
groups. 
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The paper is structured as follows. Section 2 treats the boundary components 
OO, of O, k = 1,2,...,n-—1. Section 3 is devoted to the finite-dimensional 
representation theory of the group A = GL(n,C) that acts on the cone. The 
main construct is that of a bitriangular structure (N~,D,N*) for A attached 
to a boundary component 09; by means of which we realize certain irreducible 
finite-dimensional holomorphic representations of A that are “admissible” for 
O0;. These admissible representations 7 have the property that we can define 
and analyze the operators 7(z)~! for x € Ox, despite the fact that det x = 0. 

The main theorems of the paper appear in Sections 4 through 6. Section 4 
treats the required invariant theory. In particular, we construct all of the N~- 
invariants (the vectors of “lowest D-type”) and relate the spaces of invariants 
to the boundary components 0Q;. In Section 5 we prove a number of theorems 
characterizing the image and kernel of the operators 7(x)~! for x € OQx. Finally, 
in Section 6 we apply the theorems from earlier sections to establish an important 
unitary equivalence (Theorem 6.8) for certain Hilbert spaces of vector-valued 
functions on 00x. 

Some personal remarks that place the current work in historical perspective 
are especially appropriate to a volume that honors Ray Kunze. This paper 
marks the return after a hiatus of nearly a decade, to a central theme of my 
long collaboration with Ray; namely, the interplay of invariant theory, special 
functions, and representations of classical groups. In fact, this paper may be 
seen as a sequel to the first paper we wrote [3] nearly a quarter century ago on 
operator-valued Bessel functions. In [3] we constructed new unitarizable highest 
weight representations by analytically continuing the holomorphic discrete series. 
The thrust of the current work, in this paper and others to follow, is to combine 
invariant-theoretic constructs along the lines of [4] with the analytic techniques 
in [3] to realize the remaining unitarizable highest weight representations. 

There are two other points of contact of this paper with the literature that 
should be mentioned. The special case of Theorem 6.8 in which n = 2 and k=1 
was first proved by H. Jacobsen and M. Vergne [5]. More recently, J. L. Clerc [1] 
studied vector-valued singular highest weight representations from an analytic 
point of view. His paper sparked the interest in taking a fresh look at old work. 

My closing words are for Ray — my teacher, collaborator, and friend. Suffice 
it to say that our mathematical association in particular, and our close per- 
sonal friendship in general, have been a deep source of satisfaction and joy, and 
constitute one of the high points in my professional life. 


2. Boundary Components 


2.1 The cone 2. Let C”*%” denote the algebra of all complex n x n matrices, 
and J = Jp the real subspace of all Hermitian matrices x = x*, where x* = Zz’. 
Write x > 0 to mean that the matrix x is non-negative, and x > 0 when z is 
strictly positive-definite. Then 2 = Q(n) = {x € J : x > 0} is the cone of all 
positive-definite n x n matrices over the complex field. The group A = GL(n, C) 
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of all non-singular n x n complex matrices acts on 2 by 
(1) a-z=aza* 


and this action is transitive. The unitary subgroup L = U(n) = {l€ A: ll = 
i*} is the subgroup of A fixing the identity matrix 1 = 1, € 9, and the map 
aL + x = aa* =a-1 identifies the symmetric space A/L with the cone 2. In 
short, 2 ¥ GL(n,C)/U(n). 


2.2 The boundary components 8N,. The closure of D isQ = {ze J: 
x > 0} and the boundary 00 of NisOQN =N-N={xEJ:x>0 and detr= 
0}. The boundary decomposes according to rank as 0) = Upz5 OD: where 


(1) 00, = {x € ON: rank x = k} 


is the rank k boundary component, k = 0,1,...,n—1. Of course, AQ = 0, and 
our concern in this paper is with the components for k = 1,2,--- ,n—1. If we 
block an n x n matrix in the scheme (k,n — k), then the matrices in 02; have 
the form 


© p= (2 2) 
21 £22 
in which 211 € Q(k), 222 € A(n — k), and xq, € C("—*)**, Especially important 
for our purposes is the dense open subset OQ? of ON; defined by 
(3) ONY = {x € ON, : det x11 4 O} 


The group A, acting by (2.1.1), preserves 0;; and there exists a relatively 
A-invariant measure dm(x), unique up to positive multiples, such that 


(4) dm(a- x) = | det al|?*dm(z) 


for all a € A. It can be shown that 

dx1,dxr21 
d = 

(5) m(z) (detai,)-* 


is such a measure, where dx, and dx2; denote Lebesgue measure on 2X(k) and 
C(n-*)xk | respectively. 


2.3 The orbit space U(k)\C**". Fix an integer k with 0 < k <n, and let 


(1) W =W, = {we C**” : rank w = k}. 

The subset W is dense and open in C**". We will often write an element of 
ckxn as 

(2) w = (w1, we) 


with w, € C*** and we € C*Xx(™-*). Let U = U(k) be the k x k unitary 
group, acting on C**” by left matrix multiplication. We consider the orbit space 
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U\C**" for this action, and note that U\W is dense and open in U\C**". The 
mapping 

(3) wer=w"w 


identifies the orbit space U\W with 00 ,, and the action (2.1.1) of a € A on ON, 
corresponds to right multiplication on C**” by a*. That is, 


(4) wa*ea-s 


Under this identification, Lebesgue measure on C**” corresponds to the measure 
dm({x) on 09;. That is, there exists a positive constant c such that 


(5) (x)dm(x) = cf f(w*w)dw 
OK Ckxn 
for all integrable functions f on OO,x. Finally, set 
(6) W° = {w eW: detw: #0} 
Under the mapping (3), W° corresponds to 002. In W-coordinates, the elements 
(2.2.2) in OQ; are given by 


wiw, ww 
o Zus( a ee) 
WoW) WW 
Although it is not necessary in our work to have the exact value of the constant 
c in (5), it can be computed by evaluating the integral on the Gaussian function. 
The resulting value is 


2*(1!1)(3!)--- (2k — 1)! 
(8) C= ae 





2.4 The embedding of C**” in C"*". For w € W given by (2.3.2) we 
define the n x n matrix 


(1) ut= (OF) 

This embedding of W in C”*” commutes with right multiplication; i-e., in ref- 
erence to (2.3.4) 

(2) (wa*)* = wt a* 

for w € W and a€ A. Each x € 02; decomposes as a product 

(3) z= ww = (w*)*®wF 


in which the U-orbit of w is uniquely determined by x, and the mapping 2 > w* 
identifies x with w*. Under this identification, a- xz ++ w%a*. Note that the 
matrices w* in which det w; 4 0 correspond to the subset 00% given by (2.2.3). 
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2.5 The subgroup B. Take 


0 -=(59) 


as a fixed point in 00,, 1 = 1, being the k x k identity matrix. To each 
w = (w1,wW2) € W° we associate the matrix 


(2) b= Hu) = (SEO). 


wi 
Then, as w varies through W°, the matrix 
(3) 2 =b-e=w'w = (w")*(w*) 
varies through 002. Thus, the subgroup 
(4) B= {b=d(w):w EW} 


of A acts transitively on 002. Let T be the group of all upper triangular k x k 
matrices 


tii tig +> ote 
O tag -+: tor 
(5) f= ie 5". 
Os. Ya. as. Thee 


in which t;; > 0 for all 7 = 1,--- ,k; and let Bo be the subgroup of B consisting 
of all matrices of the form 


(6) bo = Wes) = (% a 


with t € T and s € C*X("-*)_ Since each w,; € GL(k,C) can be written uniquely 
as w, = ut with u € U = U(k) and t € T, it follows that Bo acts simply 
transitively on OO. Put another way, the matrices (t,s) € C**" such that 
t € T are a complete set of representations for the orbit space U\W°. In the 
coordinates t,s of Bo, the measure (2.2.5) has the form 


(7) dm(x) = 2" tit 138-3 ---th_1 ithe dt ds 


where z = bo -e and dt and ds are Lebesgue measures on T and C**("-*), 
respectively. 
As a final remark, we observe that 


(8) w* = eb* and (w*)* = be 


where w* is given by (2.4.1) and b by (2). 
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3. ON,-admissible representations 


3.1 Algebraic induction. Let A be a complex algebraic group. A bitri- 
angular structure for A is a triple (N~, D, N*) of complex algebraic subgroups 
having the following three properties: 

(a) N+ are unipotent subgroups, each of which is normalized by D. 

(b) N~ D and N*D are semi-direct products of D with N~ and N7, respectively. 
(c) The multiplication (n~,d,n+) + a = n-dn* is a biregular mapping from 
N- x Dx Nt toa dense open subset of A. 

The method of algebraic induction [4; §6] provides an explicit construction 
of all the irreducible finite-dimensional holomorphic representations 7 of A in 
terms of certain irreducible finite-dimensional representations A of D that we 
refer to as inductive (with respect to the bitriangular structure). In particular, 
the mapping \ + 7 = 7) parametrizes the finite-dimensional algebraic dual of 
A. 

We are interested specifically in the group A = GL(n,C). A function on A 
is regular if it lies in the algebra generated by the entry functions a;; together 
with the function 1/deta; and a regular function is polynomial if it does not 
involve 1/ deta. A representation of A is algebraic if the matrix entries are regu- 
lar functions. We use the abbreviation ifdh for an irreducible finite-dimensional 
holomorphic representation of A. Such a representation is necessarily algebraic, 
so the mapping \ ++ 7 = 7) actually parametrizes the finite-dimensional holo- 
morphic dual of A. 

In its most familiar setting the bitriangular structure is “minimal”; that is, 
D is the diagonal (so-called Cartan) subgroup of A, and N+D are the opposed 
upper and lower triangular (Borel) subgroups of A. Here, algebraic induction is 
a variant of Cartan’s theorem of the highest (or lowest) weight; the irreducible 
representations \ are all one-dimensional and of the form 4 = A, where 


() Aad) = a dg? dp 


for d = diag(d),--- ,d,) € D; and A is inductive if and only if 


(2) Mm, >-+- > Mn. 


The representation 7 = 7 of A corresponding to A, is said to have signature 
m = (m1,-++,Mn), and Am is usually known as the lowest weight character of 7. 
In particular, the n-tuples m satisfying (2) parametrize the ifdh representations 
of A= GL(n,C). 

For our purposes in this paper, however, we require n—1 different bitriangular 
structures, each being maximal rather than minimal. Each of these bitriangular 
structures is associated to a boundary component 00; of N, k =1,2,---,n—1. 


3.2 The bitriangular structure for 0N,. Fix a boundary component 00x, 
0<k <n, and write an n x n matrix according to the (k,n—k) blocking scheme 
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(1) os ( 11 2) 
@21 a22 
We associate to 00, the bitriangular structure (N~,D,N*) in which N~ and 
N+ are the opposed unipotent subgroups of matrices 


(2) mO=( ‘) and nin) = (6 ”) 


respectively, where € € C("—*)** and n € Ckx("-*); and D is the diagonal block 
subgroup of matrices 


(3) a=das)=(5 5) 


with a € GL(k,C) and 6 € GL(n—k,C). Fora € A with det ay, £0, 
(4) a =n (a2); )d(ay1, @22 — @214;,'412)n* (aj a12) 


and the mapping (n~,d,n+) + a = n~dn* is seen to be biregular from N~ x 
D x Nt to the dense open subset {a € A: det ai; 4 0} of A. 


3.3 The induced representations 7). We now describe the induction 
procedure for the bitriangular structure in paragraph 3.2. We refer to [4; §6] 
for details and proofs in the general setting. Let A be an ifdh representation 
of D on the Hilbert space V = Vy, and let £ = Ly be the space of all linear 
transformations on Y. We assume that the inner product on V is chosen such 
that 


(1) A(d*) = A(d)* 
for all d € D. In light of (3.4.4), we set 
(2) A*# (a) = X(d(a11, @22 — A217; @12))! 


for alla € N~ DN?+, and say that X is inductive if \* extends from N~ DN* to 
a regular function A* : A — CL. 

Denote by P(A, A) the space of all regular functions f : A — V satisfying the 
covariance property 


(3) f(adn*) = X(d)~* f(a) 


for all (a,d,nt) € Ax Dx N*. From the density of N~ DN* in A, if 2 is 
inductive then P(A, A) ¥ 0, and the formula 


(4) (m (ao) f)(a) = f(az*a) 


for a. € A and f € P(A,A) defines an ifdh representation 7 = 7) of A. We 
say that a is the representation of A algebraically induced from X. We set Hx = 
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P(A, A), and we write H = H, when there will be no confusion. We introduce 
on H the inner product 


(5) (filfe) = [ ktila)lfala)yer** da 


for f,, fo € H, where da is Lebesgue measure. Clearly the restriction of 7 to the 
unitary subgroup L = U(n) is unitary, and by analytic continuation 


(6) m(a*) = x(a)" 


for alla € A. 

In this paper we do not use the concept of signature; however, for completeness 
we identify the representations 7, according to signature (cf., (3.1.1) et segq.). 
Write 


(7) A(d(a, B)) = A1(@) ® A2(8) 


where A; and Az are ifdh representations of GL(k,C) and GL(n — k, C), respec- 
tively, and let the signature of A, be (m,,... , ms) and that of Aq be (mz44,.-- , 
my). Then X is inductive if and only if m;, > mxz41, and the signature of 7) is 
(m1, es ,Mn). 


3.4 8N,-admissible representations. From (3.2.3) we see that the group 
D splits as the direct product D = D, D2 with D, = {d, = d,(a) =d(a,l):a€ 
GL(k,C)} and D2 = {dz = d2(8) = d(1, 8) : 8B € GL(n—k,C)}. We can identify 
D, with GL(k,C) and D2 with GL(n —k,C), and write \ according to (3.3.7) 
as the tensor product of a representation A, of GL(k, C) and a representation A» 
of GL(n — k,C). 

An ifdh representation \ of D will be called 00;,-admissible if two properties 
hold: 
(a) The restriction of X to Dz, is the identity. 
(b) The mapping a+ \(d;(a))~! is polynomial. 
Property (a) means that A2 = 1 in (3.3.7). Equivalently, 


(1) (d(a, 8)) = Ar (a) 


for all a € GL(k,C) and 8 € GL(n — k,C), and we can write V,, for V, and 
Ly, for £. From property (b) we define a polynomial mapping p : C*** 4 Ly, 
such that 


(2) (a) = Ax(a)7? 
for a € GL(k,C), and 
(3) H(a1@2) = w(a2)u(a1) 


for a1, Q@_ € CEX*, 
Note when is 0Q;-admissible, formula (3.3.2) for A* has the simple form 


(4) d* (a) = w(ai1) 
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for a € C”*”; and the defining covariance condition (3.3.3) for H, = P(A, ) 
becomes 


(5) f(ad(a, 8)n*) = ua) f(a) 


for a € A, d= d(a, 8) € D, and nt € N*. Moreover, a function f € H, extends 
uniquely from A to a polynomial function, again denoted f, on all of C"*”; and 
(5) holds for all a € C™*",a@ € CEX*, BE C(P—*)*(™—-*) and nt € Nt. 

When \ is 0Q;-admissibie, we will also say that the induced representation 
7 = Ty, defined by (3.3.4), is ON,-admissible. We mention in passing that if 
the signature of A, is (m,,...,mx), then m, > 0 and the signature of 7 is 
(mj,..-,™Mx,0,... ,0). 


4. Vectors of lowest D-type 


4.1 N~—invariants. Let 7 = 7) be O00,-admissible and set H = H,. We 
define \; by (3.4.1) and yw by (3.4.2), and we denote the space of N~—invariants 
by 


(1) H- ={f €H:n(n-)f =f for all n™ € N}. 
From (3.2.4), (3.4.5), and the density of N~ DN* in A, if f € H~ then 
(2) f(a) = wlan) f() 


for all a € A. This motivates the following characterization of H~. 


4.2 Theorem. (a) Letu€V=V),, and define fy: A— V by 


(1) fo(a) = p(air)v. 

Then fy € H-. 

(b) The mapping ® : v+> f, is an isomorphism of V with H-. 
(c) For any uv € Y, 


(2) m(n~d(a, B)) fo = frau 
for alln” € N~ andd=d(a,) € D. 


(d) There exists a positive constant c depending only upon n,k, and r,, such 
that 


(3) (fur [fu2) = e(v; |v2) 
for all v1, v2 EV. 


PROOF. From (3.2.4), the density of N~ DN* in A, and (3.4.3), it follows 
that f, satisfies the covariance property (3.4.5). Since f, is regular, we have 
shown that f, € H. Since (n~a)1; = a1; for alln~ € N~ anda € A, it follows 
that f, € H-. This proves part (a). Part (b) is a consequence of (4.1.2). Part 
(c) follows from the identity ((n~d(a, 8))~!a)11 = a~!a11. We prove (3). From 
(3.3.5) and (1), 

(for | for) = (My, v1 |v2) 
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where 
M), = f eo y(anafs da 
A 


We integrate in the variables aj2, a21, and a22 to obtain 


M, = a | ei y(ay.at; dai 
GL(k,C) 

with c, = 7”-*. Introduce polar coordinates a1; = r!/2u with u € U = U(k) 

and r = ayat, € O(k). From {3;(4.8)], there exists a constant co such that 


da; = Codudr with du normalized Haar measure on U and dr Lebesgue measure 
on 2(k). Then 


(4) My, = e120’, (k) 
where 


ry, (k) = ee e Thi (r)—ldr 


is the gamma function for the cone N(k). This integral converges absolutely to 
a scalar matrix [2;§2.4], which establishes (3) and completes the proof. 

We remark that although our work does not require the exact value of the 
constant c in (3), that constant can be readily calculated. In fact, the constants 
appearing in (4) have the following values: 


k-1 
2 2 
eq =n © and cg = Qhak(kth/2 I] j! 


j=l 
and 
k 
Ta, (k) = 27D TY mj tk 9), 
j=l 
where A, has signature (m,,--- ,m,). Hence, 
2 (m; +k— 9)! 
5 c=" iL  , 
6) II (k—j)! 


We make two remarks related to parts (c) and (d) of the theorem. First, 
formula (2) can be rephrased as the intertwining property 


(6) n(n~d(a, B))®-! = Ax (a). 


Secondly, formula (3) is equivalent to the statement that the map v + c~!/?f, 
is unitary from VY to H-. 


4.3 Definition. In light of (4.2.2) - and in analogy to the classical termi- 
nology used in the context of the minimal bitriangular structure (cf., paragraph 
3.1) — we refer to elements of H™ as vectors of lowest D-type. 
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4.4 The complementary space H‘+). Let H‘+) be the subspace of H 
spanned by the set {r(n*)f —f:f €©H,n* © N*}. If f— CH, then 
(f>[a(n*)f — f) = (n(n*)*) Ff) — FOIA) = (FIP) — (Ff) = 0 


since (nt)* € N~. Thus, H+) | H-. That is spanned by H~ and H‘*) is 
proved in [4; Theorem 6.6]. We have, then, the following important result. 


4.5 Theorem. H decomposes as the orthogonal direct sum 
(1) H=H- eH). 
4.6 The projection on H~. Since y is polynomial, from (4.2.1) and (3.3.4) 


we conclude that the mapping a — 7(a)~! is polynomial. Thus, there exists a 
polynomial map v = v,, from C”™” to linear transformations on H such that 


(1) v(a) = n(a)~ 

for a € A, and 

(2) v(a,a2) = v(a2)v(a1) 

for a1,a2 € C"*”. Recall the fixed point e € M given by (2.5.1), and set 
(3) E = v(e). 


4.7 Theorem. E is the orthogonal projection of H onto H-. 


PROOF. We need to show that Ef, = fy for all v € V, and Ef = 0 for all 
f © HO). We rewrite (4.2.2) in the form 


(1) v(n-d(a, B)) fe = fulade 


for n— € N~ and d(a, 8) € D. Since both sides are polynomial in @ and #, the 
identity (1) holds even when a and f are singular. Since e = d(1,0), we see from 
(1) that Ef, = v(d(1,0))f. = fucayo = fo. Next, we use the identity nte = e 
for all nt € NT, to see that FE = v(e) = v(nte) = v(e)v(n*) = Ev(nt). Thus, 
En(nt) = E, or equivalently E(x(n*)f — f) = 0 for all f € H, from which it 
follows that Ef =0 for all f € H“). 


5. The operators v(x) for x © 02; 


5.1 The operators v(x) for singular x. Let 7 = 7), be a OM,-admissible 
representation of A, acting in the space H = H,, and let vy = vy, be the poly- 
nomial mapping on C”*” given by (4.6.1). Since in general z is not polynomial, 
n(x) is not defined if detz = 0. However, v(x) is always defined, even when 
det x = 0. We are interested, in particular, in these operators when x € 00x, 
and we wish to calculate the kernel and image of v(x) for x € ON,. 

Recall from paragraph 2.4. that each z € 0, can be written as x = (w*)*w# 
for some w € W = {w = (w1, we) € C**" : rank w = k}. Of course, det w* = 0, 
so it is natural to first describe the kernel and image of the operators y(w*) and 
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v((w*)*). We make two simple but important preliminary observations. First, 
from (2.4.1) and (2.5.1), ew = w* for all w € C**"; hence 


(1) y(w*)E = v(w*) 


where £ is the orthogonal projection of H onto the space H™ of vectors of lowest 
D-type. Secondly, for any operator T on a Hilbert space 


(2) kernel T = (image T*)*. 


Finally, we recall from (2.3.6) that W° consists of those w = (w 1, we) € W for 
which det w, 4 0. 


5.2 Theorem. (a) For any w € W° 
(1) imagev((w*)*) =H. 


Moreover, the restriction of v((w*)*) to H~ is an isomorphism of H-. 
(b) For any w € W° 


(2) kernel v(w*) = H+) 


and the restriction of v(w*) to H™ is injective. 
(c) For anyw € W°, 


(3) image v(w*) = v(w*)H- 
and 
(4) kernel v((w*)*) = (v(w*)H-)+. 


PROOF. Part (b) follows by (5.1.2) from (1); equation (3) is immediate from 
(2); and (4) follows from (3) by (5.1.2). Thus, we are left with the proof of part 
(a). Let v € V),. We first observe from (4.7.1) that 


(5) v((w*)*) fo = Prrcwy)e 


where wy is the contragredient matrix wY = (w{)~!. Thus v((w*)*) maps H- 
isomorphically onto itself. In particular, we know that H~ C imagev((w*)*), 
and all that remains is to prove the reverse containment. Taking adjoints on 
both sides of (5.1.1), we see that Ev((w*)*) = v((w*)*), which proves that 
image(v((w*)*)) CH. 


5.3 Theorem. Let x € ON? and write x according to (2.5.3) as x = b-e with 
be B. Then 


(1) kernel v(x) = n(byH) 


(2) image v(x) = (bY )H™ 
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where bY = (b*)~} 


PROOF. Since v(x)* = v(x), (2) follows via (5.1.2) from (1). We are left with 
proving (1). Let f € H. Then f € kernelv(z) if and only if (v(x) f| f’) = 0 for 
all f’ € H. Since 
(u(x) FI’) = (u(b-e) ff’) = (u(beb") fF) = (VO) Ev(b) ff’) = VO) Ff Ev(0) f) 


we see that f € kernel Ne ) if and only if v()f 1 H™; or equivalently, v(b)f € 
H+). Since v(b) = 7(b-1), f € kernel v(x) if and only if f € 7(b)H™. 
If we combine Theorems 5.2 and 5.3 we obtain the following results. 


5.4 Corollary. Let x € ON° and write x asx = w*w withw € W°. Then 


(1) image v(x) = imagev(w*) = v(w*)H- 
and 
(2) kernel v(x) = kernel v((w*)*) = (v(w*)H7)+. 


Proor. From (5.3.1), (5.2.2), and (2.5.8), image v(x) =v(b*)H7 =u (b*)EH- 
= v(eb*)H- = v(w*)H- = imagev(w*). This proves (1), and (2) follows from 
(5.1.2). 


5.5 Corollary. Let x € 002, and view Ev(x)E as a linear transformation 
on H-. Then Ev(x)E is positive-definite. 


PROOF. Write z = w*w with w € W°, and let v € Vy, with v 4 0. Then by 
part (a) of Theorem 5.2, (Ev(x)Efu| fo) = (v(x) ful fe) = |lu((w*)*) fell? > 0. 
6. The Hilbert space L2(09,) 


6.1 The pre-Hilbert space L?,(ON)pre. We apply the results of Sec- 
tion 5 to infinite-dimensional analysis on 0Q,. Let 7 = 7),, as usual, be a 
00,-admissible representation of A, and define v by (4.6.1). Let dm(x) be the 
relatively invariant measure (2.2.5) on 00;. We denote by L? (80k) pre the space 
of all measurable functions ¢ : OQ, — H, such that 


(1) 7 __ WE)@)|0(@))am(2) < oe 
The formula 


(2) (G1142)» -[. (v(2)$1(#)|b2(2))dm(c) 


defines a non-negative sesquilinear form on L?(80x) pre which fails to be an inner 
product because there exist non-zero vectors of length 0. That is, there exist 
null vectors. Let 


(3) Ny = { € L(A) pre + |I Ally = OF 
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be the subspace of null vectors, and 
(4) L?(80K)~ = L7(OQ%)pre/ Nv 


the quotient space. Then L?(00,)~ is a Hilbert space. We want to give an 
explicit concrete realization of this Hilbert space. 


6.2 Lemma. A function ¢ € L2(O0k)pre is a null vector if and only if d(x) € 
kernel v(x) for a.e. x © OOg. 


ProoF. If ¢(x) € kernelv(x) a.e., then from (6.1.1) it follows that ¢ € N,. 
Conversely, suppose ¢ € N, and write x € ON? as x = w*w with w € W°. Then 


lel? =f (v(e)ole)jo(e))am(z) = [ Ilv((w*)*)4(a)|[?dm(x) = 0 


and we see that ¢(z) € kernel v((w#)*) for a.e. x. The lemma now follows from 
(5.4.2). 


6.3. Lemma. For each x € ON;, let H'(x) = image v(x) and let P,(z) 
be the orthogonal projection of H = H, onto H'(z). For any ¢ € L2(O0k)pre 
define the function ¢, by ¢1(z) = Pi(x)(¢(x)). Then $1 € L2(O0k)pre and 
ll — ¢1|| = 0; ue., 6 = , modulo N,. 


ProoF. Set H?(x) = kernelv(z) and let P:(x) be the projection of H onto 
H?(x). For ¢ € L2(ONK)pre, set d2(z) = Po(zx)(¢(z)). Since H = H! (x) @ H?(z) 
for each x € ONx, it follows that (x) = ¢1(x) + ¢2(z). By Lemma 6.2, do € NM, 
and ¢ = ¢, modulo N,. 


6.4. Definition. Let L?(09;,) be the space of all measurable functions ¢ : 
00, — H, such that 


(1) g(x) € image v(x) 
for a.e. x, and such that (6.1.1) holds. 


6.5 Theorem. Equipped with the inner product (6.1.2), L2(00,) is a Hilbert 
space that realizes L?(00,)~. That is, the mapping > ¢+WN, is unitary from 
iL (00%) to oy (00%). 


That L2(00,,) is a Hilbert space follows from the usual Riesz-Fisher type argu- 
ment. That L7(0,) is unitarily equivalent to L2(00,)~ is merely a rephrasing 
of Lemma 6.3. 


6.6 A unitary equivalence. Let x € ONf, and write z = w*w with w € W°. 
If ¢ € L?(O0,), then by (5.4.1) there exists a measurable function h : C**" > 
Vy, such that 


(1) (2) = v(w*) rw) 
for a.e. x. Next, let 


(2) T(w) = ww* = wywy + wew3 
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for w = (w1,w2) € W°. Evidently, r(w) is a k x k matrix that lies in the cone 
Q(k), so 7(w) is invertible. That is, r(w) € GL(k,C). From (2.4.1), 


(3) w* (w#)* = ce 2 
Then from (1), (2.4.1), and (4.2.3), 
(v(x)o(x)|6(x)) = ||e((w*)*) d(x) ||? = |v ((w*)*)v(w*) fawwy I? 
= ||v(w* (w*)*) frcwyll? = lle (72 8)) Facul” 


0 
= | far (r(wy)tacwyll? = €llAr(r(w))“*a(w) |? 
where c is the constant (4.2.5). Thus, by (2.3.5) 


(4) ae (u(x) o(z)|o(z))dm(x) = C IA (7 (w))*A(w)|[Pdw 


Ckxn 


for all ¢ € L?(00;), where C is the product of the constants (4.2.5) and (2.3.8). 
We are led, therefore, to define the following Hilbert space. 


6.7 Definition. Let L?(C**", X,) be the space of all measurable functions 
h: Cx" _,Y), such that for a.e. w € C*X” 


(1) h(uw) = Ai (u)h(w) 
for all u € U = U(k), and such that 


[ \\h(w)][? dw < 00. 
Ckxn 


Equipped with the inner product 
(2) (hala) = ff (ha(w)lhatw)) a 


L?(C**", )1) is a Hilbert space. 
The following result is a consequence of (6.6.1) and (6.6.4). 


6.8 Theorem. For h € L?(C**", \,) define y : 0N2 > H,, by 


(1) p(x) = C7? v(w*) fr, (e(w))r(w) 


where x = w*w with w € W°, andC is the product of the constants (4.2.5) and 
(2.3.8). Then the mapping h > w is a unitary equivalence of L*(C*X", A1) with 
17 (8x). 


The unitary equivalence in Theorem 6.8 is the foundation for a variety of 
results in harmonic analysis and infinite-dimensional representation theory re- 
lated to the boundary components of 2. For example, Theorem 6.8 underlies 
the relationship among operator-valued Bessel functions on the boundary com- 
ponents 00; singular highest weight representations of the group U(n,n), and 
the oscillator representation acting on L?(C**") for 0 < k <n. These ideas, and 
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their extension to symmetric cones more generally, will be the subject of future 
papers. 
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ABSTRACT. Consider the group GL(p,C) x GL(q,C) acting on @ @ CY. 
We ask: How does A*(C? @ C1) decompose under the induced action of 
GL(p,C) x GL(q,C)?_ In this paper, we determine the structure of the 
GL(p, C) x GL(q, C)-module A*(@” @ C7). 


1. Introduction 


Consider the group GL(p,C) x GL(q,C) acting on © @ C!. We ask: How 
does A*(C” @ C’) decompose under the induced action of GL(p, C) x GL(q, C)? 
Whereas the decomposition of P(C’ @ C*), the polynomial algebra on © @ C!, 
has been analyzed by many authors, little has been said about the decom- 
position of A*(C’ @ C’). In this paper, we determine the structure of the 
GL(p,C) x GL(q,C)-module A*(C’ @ C*). It is interesting to note that Lie 
algebra cohomology plays the same role in the decomposition of A*(C” @ C*) 
that Capelli identities play in the decomposition of P(C’ @ C*). In fact, there is 
a variant of Bott’s “strange equality” (equation 15.3 of [B]) proved by Kostant 
[K] that underlies the spirit of our program. We first change our perspective to 
consider a more general situation. 

Let G be a compact connected Lie group with K aclosed connected subgroup 
so that G/K is a Hermitian symmetric space with respect to the action of G. If 
T is a maximal toroidal subgroup of K, T is also a maximal toroidal subgroup 
of G. We may assume that J’, K, and G are the respective real forms of complex 
groups Tc, Ke, and Ge. If g is the Lie algebra of G and ¥ is the Lie algebra of 
K, we have the Cartan decomposition 


g=t+p,g=b +p 
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where 


pe=pytp-_, (pt, pt] =0, [pz, ps] C & and [t, ps] C ps . 


Let W(K) denote the Wey] group of K and W(G) the Weyl group of G, let N 
be a maximal nilpotent subgroup of K¢ normalized by T¢ and let H*(G/K,C) 
be the deRham cohomology of G/K. Then our form of the strange equality is 


dim H*(G/K, C) = |W(G)|/|W(K)| = dim(A*ps)” . 


The first equality has been established by several authors in several different 
ways and we will not elaborate on it. In [K], Kostant explains Bott’s strange 
equality and establishes the second equality. One of our goals will be to give 
another proof of this second equality. We now elaborate. 

Consider the space 


I, = Homx(A’ pe, C) = (ATpe)¥ . 


Since there is a Z € & such that adZ|ps = +1, we see that J, = 0 if r is odd, 
while 
Tor = Hom x (Ap, C) = Endx(A"p4) . 


Hence H?*(G/K,C) = In-, and H**+1(G/K,C) = 0. Let 7, be the set of 
irreducible holomorphic representations of Ke occurring in A’py. As a Ke- 


module 
A’p, = >, m(D)Vp , 
Dé€r, 


where Vp is the irreducible module corresponding to D occurring with mul- 
tiplicity m(D) in A’p,. Setting m = dimpz, h = dimH*(G/K,C), and 
m m 
p = dim(A*p,)*%, we have h = >> D> m(D)? and p= >> YD m(D). We 
r=0 Der, r=0 D€r, 
see that h = p, if and only if m(D) = 1 for all D and all r. (This is in fact 
Kostant’s result.) We will give a new proof of this result. 
Our method here is to appeal to the trivial inequality p < h, and thus use 
h as an a@ priori cap on the possible dimension of (A*p)%. We will in fact 
prove p = h by explicitly constructing |W(G)|/|W(K)| linearly independent 
elements of (A*p,)%. This will determine the complete Ke-module structure 
of (A*p,). Our technique involves a case by case study of each irreducible 
Hermitian symmetric space of compact type and is reminiscent of Weyl’s use of 
Capelli’s identities. 
Finally I must add that these results have been obtained independently by 
Carey Rader [R]. 


2. The classical cases 


Recall the six irreducible Hermitian symmetric spaces of compact type: 


(a) U(p +49)/U(p) x U(q), 
(b) Sp(n)/U(n), 
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(<) $O(2n)/U(n), 
(d) SO(n + 2)/SO(n) x SO(2), 
(e) E¢/Spin (10) -T, and 

(f) E7/Ee-T 


We now examine each case. 


(a) G=U(p+q), K =U(p) x U(g), |W(G)| = (p+ )!, |W(K)| = lg! 
The map of U(p) x U(q) to K given by mapping 


(A, B) > (A, ‘Bo") 


is an isomorphism. The representation of GL(p,C) x GL(q,C) on py is the 
standard action on C? @C?. Let {e1,---,e,} be the standard basis of C’, and 
let {f1,---, fg} denote the standard basis of C?. Let 


Y(p,q) = {me Z?:qg>m,>--->m, >0}. 


Consider a p x q board of squares where (i,j) stands for the square in the i**- 
row and the j**-column. For m € Y(p,q), we say (i,j) € m if j < m;. For 


m € Y(p,q) set 
um)= A (e&@fj)- 
(8,5 )Em 


Let nj (resp. nz) be the maximal nilpotent subalgebra of gl(p, C) (resp. g!(q, C)) 
consisting of strictly upper triangular matrices. Then n= nj @ Ng is a maximal 
nilpotent subalgebra of g!(p, C) @ g!(q,C). Now n; (resp. ng) is spanned by the 
p X p-matrices (resp. q x g-matrices) Ej, (resp. Fx) (k < 1) whose only nonzero 
entry is a 1 in the k**-row and I**-column. Since (k, j) € m whenever (I, j) € m 
we have E,;v(m) = 0 for Ex; € m1, and since (i,&) € m whenever (i,/) € m we 
have Fyiv(m) = 0 for Fy: € no. Thus v(m) is the highest weight vector of an 
irreducible GL(p, C) x GL(q, C)-module in A!™!p,. Note that m € Y(p, q) is the 
Young diagram of an irreducible representation of GL(p,C), and the transposed 
diagram m', formed by making the i**-column of m the i**-row of m', represents 
a Young diagram of GL(q,C). The irreducible module with highest weight vector 
v(m) is just m@m'. If Hy = diag (z1,---,zp) € gl(p,C) and Ha = diag 
(v1, ia Yq) € g!(q, C), we have 


(Hi + He)v(m) = bs mr, + > ni v(m). 
i-1 j=l 


Hence, if m,m! are in Y(p,q) and m # m’, the corresponding representations 
are inequivalent. To show that Y(p,q) gives a complete parametrization of the 
GL(p, C) x GL(q, C)-module structure of A*p4 it suffices to show that |Y(p, q)| = 
(p + q)!/p!g!. This is easily accomplished by fixing p and using induction on q. 

This answers the question posed in the introduction. 


(b) G= Sp(n), K = U(n), |W(G)|/|W(K)| = 2°. 
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The action of GL(n,C) on py is the action of GL(n,C) on $?(C”). Let 
Y(Sp(n), U(n)) = {m € Z* :n > m, > --- > my > O where mj; > m;41 
whenever m; > 0}. For each m, we say (i,j) € mifm,; > 0 andi < j <i+m,—1. 
For a given m, set 


ume «hem ee; (ei ej = 1 ej; + ej ei). 


As in (a), v(m) is the highest weight vector of an irreducible representation of 
GL(n, ©), and if m # m’, the corresponding representations are inequivalent. A 
simple calculation yields |Y(Sp(n), U(n))| = 2". Hence we have the complete 
decomposition of A*p,. 


(<) G = $O(2n), K = U(n), |W(G)|/|W(K)| = 2-1, 

As a GL(n, C)-module, p} is isomorphic to A?C". Let Y(Sp(2n), U(n)) = 
{me 2"! :n-—1> m >--- > myn-1 > 0 where mj > m;41 whenever 
m; > 0}. For m € Y(SO(2n), U(n)), say (i,j) € mifi< 7 < m,; +i and set 


v(m)= A (eAe;) (ecAe; =e; @e; —e; Be;) . 
(85 )Em 


As in (a), v(m) is the highest weight vector of an irreducible GL(n, C)-module, 
and if m # m’, the corresponding representations are inequivalent. Since 


|Y (SO(2n), U(n))| = 2"—!, we have the decomposition of A*p,. 


(d) G = SO(n+2), K = SO(n) x SO(2), |W(G)|/|W(K)| = n+2 when n is even 
and n+ 1 when n is odd. In this case the representation of Ke on py is just the 
standard representation of SO(n, C) on C” times a scalar. The representation of 
Ke on A’p x is the standard representation of SO(n, C) on A°C” times a scalar 
and so is always irreducible except for the case where 2r = n, and in this case 
the representation splits into the direct sum of two irreducible representations 
(see [J] §6). This completes our picture of the classical types. 


3. The exceptional types 


Until now the weight graph of the representation of Ke on py has not played 
an obvious role in our picture. It is virtually impossible to discuss the exceptional 
types without reference to the weight graph. Although we did not refer to the 
_ weight graph explicitly in making our calculations for the classical types, it in 
reality made our calculations possible. 


(e) G = Eg, K = Spin(10) x T, |W(G)|/|W(K)] = 27. 
The representation of Spin(10) on py is a half spin representation 


1 
0 


ree 
a 0 
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The root diagram of Spin(10) is 


1 
ye 
0--—0-——-0 


4 3 2 


and the weight graph of the representation of Spin(10) on py is 


Z™, 0 
ne a N77 NF Nua aig 
LAr 
NY 


The initial dot represents the highest weight A. The second dot is obtained by 
subtracting a, from 4, etc. A subgraph is called acceptable if whenever a dot 
is contained in the subgraph, the subgraph contains any dot p if an arrow starts 
at p and ends at g. Clearly, any acceptable subgraph corresponds to the highest 
weight vector of an irreducible representation of Spin(10) on A*p;. The degree 
of an acceptable subgraph will denote the number of dots in the subgraph, and 
an acceptable subgraph of degree r corresponds to an irreducible representation 
of Spin(10) in A’p,. Rather than list all possible acceptable subgraphs, the 
interested reader may verify the following chart, where the top number indicates 
the degree and the bottom number indicates the number of acceptable subgraphs 


of that degree, determines all acceptable subgraphs. 
0 < deg <3 or 13 <deg <16 |4< deg <7 or 9 < deg < 12 
Each dot of the weight graph represents an element of p;. The initial dot 









aaron iene ieaaeeacnarere Saeco) 


represents the highest weight vector of p,. If 0 > 0 occurs in the graph with the 
first dot representing v and the second dot representing w, then w is a nonzero 
multiple of e_;v where e_; is the ith negative simple root vector. Thus, if X is 
an acceptable subgraph consisting of r dots representing v),...,v,, the vector 
0, A---Av, is a highest weight vector of A”p;. 


(f) G = Ex, K = Es-T, |W(G)|/|W(K)| = 56. 
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The Dynkin diagram of E¢ is 








[ 
0—-———- 0-0 0 0 
1 2 3 4 5 


The representation of Eg on py is 





and the corresponding weight graph is 


eae oN 
ie Ne Py 
NAF FV ae 
S77 
7 


Again the interested reader may verify the following chart: 


aia ateioda 5 < deg < 8 or 19 < deg < 22 = 


4. Concluding Remarks 






Although the weight graph was not mentioned explicitly in the classical cases, 
it is clear that we used the weight graph extensively in all cases. It is also clear 
that the weight graph is used to obtain a first approximation to (A*V)™ for 
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a K-module V, since any acceptable subgraph determines a unique element of 
(ATV). It is somewhat fortuitous that this first order approximation succeeds 
in accomplishing our complete task in the Hermitian symmetric space case. 
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On a Mean Value Property 
for Hyperbolic Spaces 
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To Ray Kunze on his sixty-fifth birthday 


ABSTRACT. On R”, or on a Riemannian symmetric space, let h be a posi- 
tive radial function of total integral one. If f is harmonic and f *h is mean- 
ingful, then clearly f*h = f. The problem under. discussion is whether the 
converse statement holds. 


1. Introduction 


In [1] Ahern, Flores and Rudin prove the following surprising result. Let B, 
be the unit ball in C” with the Bergman metric, i.e. the standard ball model 
of Hermitian hyperbolic space. Clearly, if f is harmonic on B, with respect to 
the intrinsic metric, then so is f og for every g in the isometry group G, and it 
follows that 


(1.1) f(9(0)) = [ (fog)dy (g€G) 


where v is Lebesgue measure normalized so that v(B,) = 1. The question asked 
in [1] is whether the property (1.1) characterizes the harmonic functions belong- 
ing to L1(v), and the answer found is yes when n < 11, and no when n > 12. 

In the present article this result is generalized to all (ie. real, complex, 
quaternionic and octonionic) hyperbolic spaces and even beyond, to a class of 
harmonic spaces, with the aid of a partly new unified proof. There is also a brief 
discussion of an analogous question in Euclidean space. 
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Our main goal in looking at this problem was to try to understand the rea- 
son for the curious dependence on the dimension in the result of [1]. Without 
claiming to have been completely successful in this, we can make the following 
remarks. 

From an intrinsic geometric viewpoint the measure v is rather arbitrary; it 
can be written as hdv where dv is the G-invariant measure and h is a certain 
positive radial function with f hdv = 1. The property (1.1) then takes the form 


(1.2) feh=f 


where the star denotes convolution with respect to dv; lifting the functions to G 
this is the usual group-convolution. 

Now the problem can be posed quite in general, and makes sense for R” as well 
as for the hyperbolic spaces: Given a fixed radial function h > 0 with f hdv = 1, 
do the conditions f og € L)(hdv),(g € G) and (1.2) together imply that f is 
harmonic? A by now classical result of Furstenberg [7] says that if we require 
f € L™ then the answer is yes for any fixed Ah on any symmetric space. The 
situation is, however, quite different for L?(hdv) with 1 < p < oo. 

The solutions of (1.2) form a linear space invariant under translations by 
G. Since h is not assumed to have compact support, the known results about 
“spectral synthesis” and “spectral analysis” do not apply, but it is still reasonable 
to start looking for solutions that are eigenfunctions of the intrinsic Laplacian 
A. (As it is proved a posteriori in [1], and more generally here, in the case of the 
hyperbolic spaces these span the space of all solutions.) For such f the problem 
immediately reduces to finding a  € C, not corresponding to the eigenvalue 
0, such that A(A) = 1 (with A denoting the spherical Fourier transform of h) 
and such that there exist eigenfunctions corresponding to A in L!(hdv). When 
h is very small at infinity, every eigenfunction is in L}(hdv) and h is an entire 
function; since h does assume the value 1 (at the \ corresponding to eigenvalue 
0), it assumes it infinitely many times. It is in the cases where hf is just large 
enough so that L'(hdv) contains some but not all unbounded eigenfunctions that 
the situation becomes delicate and the dimension of the space begins to play a 
decisive role. 

It is a pleasure for the author to acknowledge his debt to S. Helgason for some 
useful discussions and for some references to the literature. 


2. A Euclidean Analogue 


This section is perhaps not too interesting for its own sake; it is included 
here because it shows in a simple case some of the phenomena that occur in the 
hyperbolic spaces. 

Denoting |x| = (30 2?)!/? for x € R”, we fix the function 


(2.1) h(x) = cne 2717! 
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where c, = 1("+1)/2/T(241) is chosen so that fh(x)dz = 1. We note that, 

for fixed y, h(x + y)/h(x) is bounded in 2; hence, if f € L'(h(x)dz) then also 

fog € L'(h(x)dz) for all g in the Euclidean motion group G. Clearly, the 

property {(fog)h = f(g(0)), (g € G) is equivalent to the equation f *h = f. 
We denote by A the Euclidean Laplacian, 


6? 
AS Dost 


PROPOSITION 2.1. Leth be as in (2.1). The equation f xh = f has solutions 
f £0 satisfying f € L'(h(x)dx) and 


(2.2) Af =—4n?’f 
with some \ £0 if and only ifn > 9. 


ProoF. If such a solution f exists, then translating it to be non-zero at 0 
and averaging it over the rotation group K we may assume that it is radial. As 
is well-known, the only radial solution of (2.2) is (up to a constant factor) 


ina) = f emMol ae 
k 


where e, = (1,0,...,0). From the asymptotic behavior of Bessel functions (4, 
vol. II, 7.13 (3)] it follows that, for large |z], 


jr (x) ny cfa|—(t41)/2 e2lim Al |x] 
From this it is clear that j, € L?(h(«x)dz) if and only if |ImA| < 2 In fact, this 
can also be seen by first proving, by elementary arguments, that j, is in L? if 
and only if e2™//™Al(eil2) is, and then using the Schwarz inequality. 

By the general version of the Godement mean value theorem [9, p. 402] every 
solution f of (2.2) satisfies 


: fle + ky)dk = ja(y) (2) 


for all z,y € R”. Multiplying by h(y)(= h(—y)) and integrating with respect to 
y we obtain 

fxh=h(re1)f 
where h is the Fourier transform of h. Explicitly, 


A(Ae1) = (1 +A2) (0? 


with that branch of the square root which is positive on the positive half-line. 
In view of these remarks the existence of an f as in the statement of the 
proposition is equivalent to the existence of a A 4 0 with |ImA| < 1 and such 
that (1+ 2)-"# = 1. 
The range of 1 + A? for |[mA| < 1 is the interior of the parabola Rez > 
(31m z)*, and the question is whether this contains any point z 4 1 such that 
zin+1)/2 — 1. It is clear that if such a z exists at all, then e47/("+)) is such a 
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z. Now this point is inside our parabola if and only if n > 9. This finishes the 
proof. 


REMARKS. If in Proposition 2.1 we replace L} by L? for some 1 < p < ov, 
then A is further restricted, and n has to be larger for non-trivial solutions to 
exist. It is clear that for any fixed 1 < p < oo the number of As giving solutions 
tends to infinity with n. It is easy and instructive to see what happens if we 
change h to h(x) = ell”. This h is much too small: For every \ € C, jy is in 
every L?(h(x)dz) if p < 00, and the Fourier transform h(\e,) = e~”” is equal 
to 1 for infinitely many values of 4, independently of the dimension. 

We do not pursue the Euclidean case any further; in particular we leave open 
the question whether Proposition 2.1 remains true without the condition (2.2). 


3. The Hyperbolic Spaces 


The hyperbolic spaces are exactly the Riemannian symmetric spaces of non- 
compact type having rank one. As proved by Takeuchi [13], they all have canon- 
ical realizations as a unit ball B,, in some R™. They are explicitly described in 
[11]; another way to construct them is indicated in [2]. 

We fix a symmetric space X of rank one and, using the standard notations 
(cf. e.g. [9]) we write G for the connected isometry group, o for a chosen base 
point, K for the stabilizer of o , {a,} for a one-parameter group of transvections 
based at o. Then {a,0} is a geodesic line and every point in the space appears 
in the form ka,- 0 (k € K,t © R). The space has two important invariants, p 
and q, the multiplicities of the short resp. long restricted roots. (Actually, real 
hyperbolic space fits into the picture in two ways: with either p = 0 or q = 0. 
We will always consider it in the version p = 0; this gives in our later setup the 
Poincaré model, while g = 0 would give the less popular Cayley-Klein model.) 
The real dimension of the space is 


m=pt+qtl, 


another important quantity is given by 


BG ia 
The canonical realization of X is given by the map ka,o +> (tanht)k-e, where 
€, = (1,0,... ,0) € R™ and k acts on R™ in the same way as on the tangent 


space of X at o. The intrinsic volume element of X is known to be [5],[10] 
(3.1) dv = 6(t)dtdk = (sinht)?*4(cosht)‘dtdk 


where dk is normalized Haar measure on K. (We have chosen a simple normal- 
ization of dv; the literature is not uniform on this point.) 
In the canonical ball realization, with r = |x| = tanht this becomes 


(3.2) dy = (1—17?)-@+De™—-lardk = (1 — |x|?)~ +) dv(z) 
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where v is Lebesgue measure normalized to give v(B,,) = 1. Hence, in terms of 
the coordinate t we can write 


(3.3) dv = m(cosh t)~?(°*)) dv. 


Now, generalizing [1] and using the Euler beta function, we define for k > —1 
(not necessarily an integer), 


84) (TN0-) = eee Ty, O-leP Mon dete) 


for f € L!(Bm,v). (When the same B,, is the model for two different symmetric 
spaces this formula is still the same for both; what is different is the way in 
which G acts). To rewrite T;, in terms of the intrinsic coordinates we define, for 
k>-l, 


(3.5) hy (kay - 0) = (cosh g) tet tte), 


Ms 
B(¥,k+1) 
We have fh,dv =1, dv = hodv, and we can write 
(3.6) Tif = f «he 


for f € L'(hodv). We note that, since $e’ < cosht < e! and since t is the geodesic 
distance, ho(g~!y)/ho(y) is bounded on X for fixed g € G. Therefore the integral 
in (3.5) (and in (3.6)) is well defined. 

We write A for the invariant Laplacian of X, and we note that [8],[5],[10] 
for radial functions written in terms of t the effect of A can be computed by 
applying the ordinary differential operator 
d? 
dt? 

PROPOSITION 3.1. The equation f x*ho = f has non-zero solutions satisfying 
f € L'(hodv) and 


(3.9) Af =uf 


with some complex u # 0 if and only if p > 12. 


(3.8) + {(p + q)cotht + gtanht} ‘. 


ProorF. Writing u = —(? + p”), it is well known that (3.9) has a unique K 
- invariant solution having value 1 at 0, the spherical function ¢)(= ¢_,). It is 
known ((8], also [5],[10]) that },(a;-0) = F(a,b;c;z) where z = — sinh’t and 
ptir p—tr m 
Od ae ot 8 
Since we are working with the ball model where the natural variable is c = tanht, 


it may be nicer to rewrite this, using r? = z/(z — 1) and a Kummer relation, in 
the form 








(3.10) x(a - 0) = (1—1?)* F(a, c — 6; c;r7). 
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Now the solution space of (3.9) contains elements in L!(hgdv) if and only if 
ox € L1(hodv). By (3.2) this happens if and only if 


1 
/ (1 — r?)*|F(a,¢— b;¢;r?)|\r"™ "dr < 00. 
0 


By (4, Vol. I, p. 76, (9)], for large z,F(a,b;c;z) ~ Ayz7* + Aoz~? (with a 
logarithmic factor if a— 6 is an integer). Using r? = z/(z—1) this gives that our 
integral is convergent exactly when Rea, Reb > —1, i.e. when 


(3.11) [Im A| < pt+2. 


(By similar arguments it is easy to see that ¢, € L?(h;,dv) if and only if |ImA| < 
p+2(1+k)/p.) 

By the generalized Godement mean value theorem [9, p.402] the solutions of 
(3.9) satisfy the identity 


| flak nak = ox)fta-o). 


(We need the following only for k = 0, but with the same effort, in order to hint 
at some generalizations, we do the general case:) If f is also in L}(hydv), we 
multiply the identity by hz(y)(= he (y~1)) and integrate over X to obtain, for 
[Im >| < p+2(1+k)/p, 


(3.12) (f *h)(g-0) = he(A) f(g - 0). 
Here hy, is the spherical Fourier transform; using (3.10),(3.2) and the substitution 
xz =r’ we have 


hala) = | helw)oxtoddota 
D) bt 


= 7 +1) | zz 3(1—2)*+*F (a,c — b;¢;x)dz 


Be 
To compute the integral we use [4, Vol. I, p. 78, 2.4 (2)]: 
: T(s)I'(e — 
| a®l(1—2)° 1 F(a, b'; s, 2z)dx = Se mae b’; c'; z) 
0 


with s = $,b’ =c—b,c’ =k+a+ 3 -1,0<z <1. (The formula applies, since 
Rec’, Res > 0 by the condition on 4.) Letting z — 1 and using 
T'(c’)P'(c' — a— b’) 
['(c! — a)I'(c’ — b) 
(cf. [4, Vol. I, p. 104, (46)]) we finally obtain 
‘ T(k+1+ S28 )r(k+1+ 5% 
(3.13) hy (A) = ( ans 2 ) 
T(kK+1)0(kK+1+ ) 


F(a, 0'3¢’31) = 





(Another way to evaluate h,(A) is sketched in [10, p. 70].) 
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In view of (3.12) all that is needed to prove our proposition is to decide whether 
ho(A) assumes the value 1 for some 4 # +ip,|Im | < p+ 2. Now the function 
Ms 13) (for k = 0) is carried into the function ¢,(@) of [1] by the substitution 

a SG +%,)/2, and [1, Prop. 3.8] gives the answer, yes if > 12 and no if 

e, 


REMARK. The proof also shows that the possible y's in (3.9) always form a 
discrete set, namely the image under the map \ ++ —(A? + p) of the zeroes of 
the function ho(A) — 1 contained in the strip |ImA| < p+ 2. 


THEOREM 3.2. The equation f * ho = f has non-harmonic solutions in 
I} (hodv) if and only if p > 12. 


Proor. Our proof follows [1]; it is only necessary to establish that the main 
ingredients remain true in our more general situation. 

First we must see that the operators T;,(k > 0) are bounded on L!(hodv). By 
the inequality 


ines = [| [seyret *a)dy 
< | aay [maw 2dho(a)de) I) hou) 


this will follow if we can prove that 





(3.14) ho *hk < Mho 


with some constant M. (Note that G is unimodular and the functions are even.) 
For this we use the expression for the convolution of radial functions from [6] 
(cf. also [10, p. 59]), which gives 


(ho*he) (az - 0) 


(3.15) ion ia —2(p+1) —2(pt+k+1) 
= i (coshr)~*°+)( cosh s)~?°t*#+) K(r, s, t)6(r)6(s)dsdt 
0 Jo 


with K(r,s,t) =0 unless |r — s| < t < r+, in which case 


( cosh r cosh s cosht)?~! 


K t= 
(r, 8,4) ( sinhr sinhs sinh t)P+9-1 


K'(r, s,t) 





with a bounded function K’. 

Since our functions are positive, in order to prove (3.14) it is enough to es- 
timate (3.15) for, say, t > 1. In that case ( sinht)~! < je~‘; using also that 
sinh r < coshr, er < cosh r < e”, we see that (3.15) is majorized by 


cee ff e (P+2)r—(p+2+2k)s nds, 
|r—s|<t<r+s 


The integral is easy to compute by the variable change u = r+s,v =r-—s, and 
gives a majorization by C’e~?(°+1)t, which implies (3.14). 
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The next important fact is that the Tjs all commute and that the identity 
(3.16) AT, = 4(9 +k +1)(K+1)(Th — Th41) 


holds. Since the radial functions on a symmetric space form a commutative 
convolution algebra and A commutes with radial convolutions, for this it suffices 
to prove that 

Ah, = 4(9 +k + 1)(K +1) (he — het). 


This is an easy computation using (3.8). 
We only sketch the rest of the proof which contains nothing new. Repeated 
application of (3.16) gives 


(3.17) Tk f = Grp(A)Tof 


with the polynomial 


(3.18) Gkp(u) = I[a- aia 


per ee 


The subspace M = {f € L'(hodv)|Tof = f} is closed in L!(hodv); this can 
be seen by observing that by (3.14) Tp is a bounded operator from L}(hgdv) 
to L!(h,dv). The operator A preserves M, and its restriction, to be denoted by 
Aw, is bounded on M by (3.16). Writing G,() for the infinite product obtained 
from (3.18) by letting k — oo, it follows that 


(3.19) G,(Am) =TIny, 


the identity map. 

By the product expansion of the gamma function we have G,(—(A? + p”)) = 
hae), We write 2, for the image of the strip |Im A| < p+ 2 under the map 
A+» —(A? + p?) and E, for the subset of 2, where G,(j) = 1. By the remark 
after the proof of Proposition 3.1, E, is exactly the point spectrum of Ay. 

Writing @ for the monic polynomial whose zeroes are the points of E,, there 
exists an entire function H such that HQ = G, — 1, and we have H(y) # 0 at 
every js € E,. Since H has no zero on the point spectrum of Aj,, by the spectral 
mapping theorem 0 is not in the point spectrum of H(Aj,), which is therefore a 
one-to-one operator. (3.19) shows that H(Aj,)Q(Ajys) = 0, and it follows that 
Q(Aar) = 0. [1, Lemma 4.1] can now be applied and shows that M is the (finite) 
direct sum of the eigenspaces of Ay;. By our Proposition 3.1, this finishes the 
proof. 


REMARK. The same methods work if one wishes to study the solutions of 
Tf = f for some k # 0, or if one looks for solutions in some L?(1 < p < oo) 
instead of L!. In particular, all the statements of the main theorem in [1] remain 
true after replacing n by p. 
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4. Further Generalizations 


In [3] E. Damek and F. Ricci construct a large family of examples of harmonic 
spaces which all have natural realizations as unit balls in some R™. They contain 
the hyperbolic spaces as special cases. As Z. Szabé [12] proved, on any harmonic 
space the integral kernels K (p,q) depending only on the distance of p and q form 
a commutative algebra under composition; also, the Laplacian A commutes with 
the natural radialization operator. On the Damek-Ricci examples there acts, in 
addition, a simply transitive group of isometries, and the commutative algebra 
of kernels appears as the convolution algebra of radial functions on this group. 

These examples are still characterized by two non-negative integers p and q 
as in the symmetric case; the dimension is still m = p+q+1 and the invariant 
p = § +4 (denoted in [3] by Q, and known to be an integer when q # 0) still 
plays a crucial role. The geodesics through 0 are the diameters parametrized 
by tanh ¢, the Riemannian volume element and the spherical functions have the 
same expressions in terms of p and q as in the symmetric case. 

As long as p is an integer, our results apply unchanged to the Damek-Ricci 
spaces. The proofs are the same, with averaging over K replaced by the radial- 
ization operator (denoted by z in [3]), and with somewhat more attention paid 
to writing convolutions since the underlying group is not unimodular. 

The number p is only a half-integer when g = 0 and m is even. This is the 
case of the Cayley-Klein model of m-dimensional real hyperbolic space which we 
excluded from consideration in section 3. Actually, our arguments still work, but 
in order to find for exactly what values of p the equation Tj f = f will have non- 
harmonic solution, it is necessary to make a study of the function G, analogous 
to Propositions 3.7 and 3.8 in {1] for non-integer p. 

Another generalization of our results is possible in the context of the “Jacobi 
hypergroups” introduced by M. Flensted-Jensen and T. Koornwinder [5], [6],[10]. 
Here there is still a (generalized) convolution algebra, characterized by two ar- 
bitrary positive numbers p and q, and a differential operator generalizing A. 
Our general considerations remain true, and the question about the existence 
of non-trivial solutions of To f = f reduces to the study of the function G, as 
before. 
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Inversion of an Integral Transform 
and Ladder Representations of U(1,q) 


JOHN D. LORCH AND LISA A. MANTINI 


ABSTRACT. An integral transform for G = U(1,q) is studied. The trans- 
form maps the positive spin ladder representations of G on a Bargmann- 
Segal-Fock space F+% into a space of polynomial-valued functions on the 
bounded realization BY of G/K. An inversion is given for the transform 
and unitary structures are given for the geometric realization of the positive 
spin ladder representations over G/K. 


0. Introduction 


The study of representation theory of linear reductive Lie groups is intimately 
related to the study of differential equations and the physics that these equations 
model. Such are the beginnings of the theory concerning the ladder representa- 
tions of U(p, q) given in this paper. 

In [7], Jakobsen and Vergne show the unitarity of a cleverly constructed mul- 
tiplier representation 7, of U(2,2), the 4-1 cover of the conformal group, on a 
Hilbert space H, of holomorphic functions on U(2, 2)/K realized as the general- 
ized upper half plane. The elements of 1, satisfy the mass 0, spin s equations, 
and the restriction of 7, to the Poincaré group is the representation of mass 0, 
spin s. 

A previous result that is recurrent throughout [7] is a theorem of Kunze 
(see [9]) that guarantees the unitarity of certain multiplier representations. The 
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key is that these representation spaces of functions possess a positive definite 
reproducing kernel that behaves well with respect to the action of the maximal 
compact subgroup on the image space of these functions. 

Meanwhile, the oscillator representation of U(p,q) has been studied and de- 
composed into irreducible ladder representations in [2], [3], [8], and [16] to name 
a few. Consider the realization in [3]. Here the oscillator representation o is 
constructed via the Stone-von Neumann theorem on a Fock space F of (p, q)- 
holomorphic functions. We use this realization of o in this paper, despite the 
geometric advantages of the realization given in [2]. 

The relationship between the representations 7, and the oscillator representa- 
tion of U(2,2) noted in [7] is generalized by Mantini in [10]. It is in [10] that we 
see the positive spin ladder representations F,, of U(p, q) realized via an integral 
transform ®,, as a subspace S,, of sections of a homogeneous holomorphic vector 
bundle E,, over U(p,q)/ in its model as a generalized unit disk D. The fiber 
of E,, over any point ¢ € D is essentially the space P(n, C%) of antiholomor- 
phic polynomials homogeneous of degree n in q complex variables. If the real 
rank of U(p,q) is larger than one, or if the rank of Ey is larger than one, these 
sections satisfy certain linear partial differential equations. In the U(2, 2) case, 
these equations are the massless field equations and the natural action on S, 
is equivalent to the multiplier representation of [7]. The transform ©, bears 
many similarities to the Penrose transform (see [12] and [17]). We will discuss a 
version of ®, more carefully in section 2 below. There also exists an analogous 
construction of the negative spin ladder representations (see {11]), but here we 
restrict to the positive spin case. 

Since ®, is one-to-one on F, for n > 0, ®, is invertible. In section 3, an 
inverse for ®, in the case when G = U(1,q) is presented. Trailing along behind 
the inversion of ®,, is the machinery that allows the production of a Hilbert space 
structure on S, so that S, is unitarily equivalent to F, via ®,. Such structures 
are exhibited in section 4 for this case. 

Motivation for our results is essentially two-fold. On one hand, we consider the 
results of Davidson [3] (which again depend critically upon results of Kunze [9]). 
He gives natural unitary norms on U(p,q)/K for discrete series representations 
of U(p,q). These structures take on the form 


Ge E (QC OL (0), flO) auld); 


where f; € H(D,A) are polynomial-valued functions on the disk, dy is the 
U(p,q)-invariant measure on D, and Q) is the reproducing kernel for H(D, A). 
The inner product (-,-) consists of an integral with respect to the appropri- 
ate Gaussian measure. For the ladder representations of U(p,q) which are not 
square-integrable, an analogous norm does not converge (see Lemma 4.2 of [3}). 

On the other hand, we are also inspired by inversion formulas for the Radon 
transform, as given in [5]. Let P” denote the space of all hyperplanes in R”, 
and let f be a function on R” that is integrable on each element € of P” with 
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respect to Euclidean measure dmg on €. The Radon transform f of f is then 
given by 


f= [ fla) dme(se), 


while the dual transform for continuous functions y on P” is given by 


we) = [ (6) dusl®) 
where £, = {€ € P" | x € €} and dy, is the measure of total mass one on €, that 
is invariant under rotation about z. A Schwartz function f can be recovered 
from its Radon transform via the formula 


cf =L°F ((f)), 


where c is a constant and CL is the Laplacian on R”. If n is even, £*2 is defined 
in terms of Riesz potentials (see Lemma 2.35 of [5]). Operators playing a role 
similar to these fractional powers of the Laplacian appear in our results. 

In large part, we achieve our results by constructing two special operators on 
O(D, P(n, C%)), P and L. Let ¢ be the image of a monomial in F, under ©). 
Fixing ¢ € D and considering ¢(¢) as an element of P(n,C%), the mapping P is 
a projection operator such that P¢(C) is the nonzero summand of ¢(C¢) of highest 
possible weight with respect to the K action on P(n,C%). The mapping L is 
simply a weighted version of the mapping P which enables the overall inversion 
and inner products to become unitary. Using this mapping L, we produce an 
inversion formula for ®,, and the desired unitary structures. These have the form 


(@s9)(2) = tim le [fh woytecsapetreH* eM dm(oyam(e)], 


and 


(1,60) = jim Fle f (orytec, 4016.) amo), 


respectively. 

Alternative unitary structures for most of the ladder representations of U(p, q) 
are given by Rawnsley, Schmid, and Wolf (see section 13 of [14]). These repre- 
sentations occur in L?-cohomology of certain homogeneous, holomorphic vector 
bundles over U(p, q)/(U(1) x U(p — 1, q)). 

The authors would like to thank M. Davidson, E. Dunne, E. Grinberg, and 
R. Zierau for helpful conversations. We are grateful to the referee for some 
useful remarks. Finally, we would like to thank the editors of this volume for 
their efforts on behalf of the authors as well as for their patience and support. 


1. Preliminaries 


In this section we wish to gain familiarity with the oscillator representation 
of U(p, q) realized on a certain Fock space. The generalized unit disk and an 
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associated representation will be defined. We first introduce the multi-index 
notation that will be prevalent throughout the paper. 


1.1. 


NOTATION. 


For r € N, let No denote the set of nonnegative integers and let NG 
denote the set of r-tuples of nonnegative integers. 

If m = (m1, me,...,m,) € NG, then m! := mi!me!---m,! and |m| := 
m+ m2+--:+m,. If z € C’, then z™ = 2)" 25"? ++ 2m. 

Leta € Nj, 1 <s <r, and let J = (1, jo,.-., js) € N§ be such that 1 < 


F< jy << jy STs Then, if 2 6 OC", 47: 3= (25552955 S52 427, |zy|? : 
lzs1? + lz? +--+ + [zj.[?, and 25 = 22292 238 


Let R, := (1,2,...,p) and Spq:= (ptlpt eee q). When there 
is no danger of confusion, we will let R and S represent R, and S, 4, 
respectively. 

For n € No and gEN, let Nj(n) := {7 € NG | |n| = n}, and let d(n, q) 
denote the cardinality of N3(n). Note that d(n,q) = dim P(n, C4). 
The symbol ~ will denote lexicographic, or dictionary, order on Nj(n). 
Here a ~ £3 if there exists 7 € {1,...,q} such that a; = 6; fori < j and 
a; < f;. In this case we say that @ succeeds a. 

We have a partial order < on Nj given by a < £ if a; 
{1,...,q}. 

Let 7 := (0,...,0,1,0,...,0) € Nj, where the support lies in the j-th 
component of «7. We will write €; for «7 when there is no danger of 
confusion. 

For p,q € N, we let M?*7(C) denote the set of p x q matrices with 
complex entries. If X € M”*4(C), the conjugate transpose of X is 
written as X*. 


< @; fori € 


Recall that for p,q € N, U(p,q) = {9 € GL(p+4,C) | 9*Ip,q9 = In,q}, where 
Ip,q is the element of M(?+#)*(?+9)(C) given by 


I 0 
m= (5 -1,) 


If g€ G= U (p,q), the matrix g is occasionally written in block form as 


_(A B 
I=\o Dp}? 


where A and D are p x p and q x q matrices, respectively. In addition, the 
subgroup K ~ U(p) x U(q) of U(p,q) given by 


ates | A € U(p), Dev(q)} 


is a maximal compact subgroup. 
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Let dm(z) denote the normalization of Lebesgue measure on C” given by 


1 
dm(z) := —dz dy, ...dz,dy,. 
nT 
The following lemmas will be useful. 
1.2. LEMMA. Let k,l € No, and let 6. = 1 ifk =1 and 6, = 0 otherwise. 
Then 


| zhzte le!” dm(z) = 6x1 ki. 
Cc 


1.3. LEMMA. Suppose that U € C%*? is such that U + U* > 0, and ¢ is a 
holomorphic function of z € C4 for which f(z) exp(—z*Uz) dm(z) converges 


absolutely. Then 
—2z*Uz _ (0) 
BE d(z)e dm(z) = det" 


1.4. LEMMA. Suppose thatq € N and w €C. The series 


Ss 


meNé 





wil 
|m|9 
converges if |w| <1 and diverges if |w| > 1. 


Finally, we say that a complex-valued function f on C?*9 is (p,q)-holomorphic 
if f is holomorphic in zg and antiholomorphic in zg. 


1.1. The Oscillator Representation of U(p,q). Consider the Bargmann- 
Segal-Fock space 


FP4 = {f :C?t4-+C | f is (p, q)-holomorphic 
1.1 
v2) and / |f(z)|2e7 2” dm(z) < oo}. 
CPt 
By polarizing the integral in (1.1), one obtains the inner product for the Hilbert 
space F?:7, which will be denoted by (-,-). Let f € F?? and suppose 


f(z)= ae anes: 


lene 
meNG 
It follows from Lemma 1.2 that 
(1.2) (ff) = S> Jarm|? lm. 
tenh 
meNé 


As an application of Lemma 1.3, we obtain the reproducing formula for f € F?*9 
given by 


(1.3) f= ss f(w)ewheRt25us e—lwl? dm(w). 


We now define a version of the oscillator representation o of G. 
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1.5. DEFINITION. For g = © a € G and f € ¥”4, define a(g)f by 


[o(9) f|(z) = det(D*) i f(g we 219 wl’ ewhentzus e214" dm(w). 


Cpt+49 


It is not immediate that o defines a unitary representation of G on F?. 
Nonetheless we have the following theorem, due to Blattner and Rawnsley. 


1.6. THEOREM [2]. The map o of Definition 1.5 is a unitary representation 
of G on FP4, 


Now, for n € Z, let 
FP = {f © FP1| f(e-z) = a” f(z),0 € R}. 
A short computation shows that F?? is a G-invariant subspace of F?:%. For 
9 € G, let on(g) denote the restriction of o(g) to FP. 


1.7. THEOREM [16]. Let n € Z. The representation o, of G on FP is 
irreducible, and there is an orthogonal direct sum decomposition 


FRESE Fe. 


neZ 


REMARK. The representations o, are often referred to as ladder representa- 
tions, since the highest weights of their K-types lie equally spaced along a ray 
in the weight lattice. 


1.2. The Generalized Unit Disk. For p,q € N, let C?? denote C?t? 
endowed with the hermitian form h with matrix I,,,. A subspace V of C?? is 
negative if the restriction of h to V is negative definite. Let M;, denote the set 
of all negative q-planes in C”?. The generalized unit disk D,,, is given by 


(1.4) Dp,q = {¢ € M?P*7(C) | I, = C*C > O}. 


1.8. PROPOSITION. The mapping ¢++ V(¢) of Dp,q into M,, where 


V(¢) =col span FE 
q 


gives a parameterization of MZ, by Dy,q- 


PROOF. The mapping is clearly well-defined and injective into the space of q- 
planes. The g-plane V(¢) associated to ¢ € Dp, is negative due to the positivity 
condition in (1.4). It remains to show that the mapping is surjective. Let 

U 
V =col span ea € Mo 


where U, and U2 are p x q and q x gq matrices, respectively. Now Uz must be 
nonsingular, as otherwise V will admit positive vectors. Hence we can change 
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basis to see that V = V(U,Uz'). Since V is negative, U,U;* satisfies the 
conditions of (1.4), therefore U,Uy'¢€D,,. @ 
The natural left action of G on M,, given by 


ern (0 | i }) 


is transitive, and the stabilizer of V(0) is A. By Proposition 1.8, D,,, is diffeo- 
morphic to G/K and there is a natural action of G on D,,, that is stabilized at 
0¢€D,,., by K. This action is given by 


9-6 = (AC + B)(CO+ D)“, 


B 


C D 
Note that D,,, is a bounded domain in C??. In fact, it is one of the four 


classical domains (cf. [6] or [13]). In the case we are concerned with, that is 
p= 1, D,,, is simply the unit ball B? in C%. 

We will rely on our ability to integrate over B?, and toward that end we use 
the following result. For details, see [15]. 


where g = ) € Gand ¢€ Dag. 


1.9. LEMMA. Let n,y € N@ and suppose that |n| =n and |y| =m. Then 


[L stram(c) = bn (ES), 


where 6, =1 ifn =v and 6,,, =0 otherwise. 


Let P(n,C%) denote the set of antiholomorphic polynomials on C% that are 
homogeneous of degree n and let O(D,,4, P(n, C%)) denote the set of functions 
holomorphic on D,,, taking values in P(n,C?). We conclude this section with 


the construction of a representation w, of G on O(D,4,P(n,C?)). Define, for 
n> 0, a mapping J, : G x D,,, > GL(P(n, C%)) by 


(1.5) In(9,0) f(v) = det[C¢ + D] f([C¢ + D}*v) 
C : ,¢ €D,q and v € C?. It is easily shown that J, is a factor 
of automorphy. That is, if 91,92 € G and ¢ € D,, then J,(/,¢) = Id and 
Jn(9192,€) = In(915 92-¢) 2 Jn(g2, ¢). 


1.10. DEFINITION. Suppose g € G and ¢ € O(D,4,P(n,C%)). Define 
Wn(g)d by 


for g = G 


(wn(9)b)(C,v) = In(97", 6) *O(g7 Gv) 
for ¢ € Dz, and v € C?. 


Note that w, is a multiplier representation of the type discussed in [9]. This 
representation corresponds to the natural geometric action of G on the vector 
bundle E,,. In the next section we will intertwine o, and w, via an integral 
transform. 
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2. The Integral Transform 


In [10], Mantini constructs an integral transform ©®, for each n € Z, n > 0. 
We may think of ®, as a mapping of F?:? into O(D»,4, P(n, C*)). This transform 
is analogous to the Penrose transform (see [12]). In case either the real rank of 
G is larger than one or the dimension of P(n, C%) is larger than one (i.e., when 
n > 0 and q > 1), the elements in the image of ®, satisfy certain linear partial 
differential equations. These equations generalize the massless field equations of 
mathematical physics which hold when G = U(2,2) (see [10] or [7]). The image 
of the map ®,, in [10] is in a space of sections of a holomorphic, homogeneous 
vector bundle on which the group acts by the natural, geometric action. Thus 
®, implicitly gives a geometric construction of the ladder representations o,, 
although the inner product structure on this realization has previously been 
unknown. 

At this point let us recall that our primary goal is to create an inversion for 
®, and then use this inversion to develop unitary structures for the geometric 
realization of o, obtained via ®,. In this section we familiarize ourselves with 
®,, and some of its important properties. Further details may be found in [10] 
and [11]. 


2.1. PROPOSITION [10]. Suppose n > 0 and f € F?:4. Then we have an 
integral transform ®, : FP? + O(Dyq, P(n, C%)) given by 


(@,f)(¢,0) = L _HGu, ude eM” am(w), 


where v € C4 and (®,f)(¢,v) depends holomorphically on ¢ € Dpgq- 


The transform of Proposition 2.1 is simplified from the transform constructed 
by Mantini in [10]. Here we have used the trivialization of the vector bundle 
in [10] with the multiplier action w,. We have also used Davidson’s model 
of the Fock space representation a, instead of the L?-cohomology realization of 
Blattner and Rawnsley (see [2]). The transform is essentially given by restriction 
to a negative q-plane determined by ¢ € D,,, followed by a projection operator, 
giving rise to an L? version of the Penrose transform. Notice that the projection 
operator is the specialization to signature (0, q) of the reproducing formula given 
in (1.3). 

The first important feature of ®, is that it serves as an intertwining map 
between the representations o, and wy. 


2.2. THEOREM [10]. The mapping ©, is one-to-one for n > 0 and identically 
zero forn <0. Furthermore, for all f € F?4, g € G, andn > 0, the mapping 
®,, satisfies 

Wn(g)(Onf) = On(on(9)f)- 


Theorem 2.2 implies that for n > 0, ®, has an inverse defined on ®,(F2'9). 
Also, it says that o, is realized on ®,(F?:7) by a subrepresentation of wy. 
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2.3. LEMMA [10]. Fin € Z and € € Dy,g. The mapping of FP? into FP4 
given by 
fr ®, f(C,-) 


is continuous. 


The next thing to note is that for many values of p and g, the elements of 
®,,(F?7) satisfy certain interesting differential equations. Using the notation of 
[4], let O,,¢ be the (p + 1) x q matrix of differential operators where 


of 
Giz? 
(Oh ,a) iz a 5 


a> fort=ptlandl<j<q. 


forl<i<pandl<j<4q, 


2.4. THEOREM [10]. 


(a) Let p,q € N be such that both p and q are greater than one. Then 
©,,(FP%) is a subset of O(Dp,q,P(n, C%)) annihilated by the differential 
operators formed by taking 2 x 2-minors of Op. 

(b) Ifp=1,n>1 andgq>1, then ®,(F}%) is a subset of O(B4, P(n, C?)) 
that is annihilated by the differential operators formed by taking 2 x 2- 
minors of U1 ,q- 


REMARK. Incase p = q = 2, the differential operators formed by taking 2 x 2- 
minors of O22 are the complexifications of the massless field equations. Also, it 
has been shown in [4] that the K-finite vectors in O(D,,4, P(n, C?)) with respect 
to wy, that satisfy the differential equations mentioned in the theorem are exactly 
the K-finite vectors in ®, (FP). 

Suppose that n € No and ¢ € O(D,,4, P(n, C?)). Then there exists, for each 
n € N§(n), a holomorphic complex-valued function 7” on D,,, such that 


(2.1) HG») = D> vM(C) 


neEN3(n) 


The component functions 7) in (2.1) are indexed by n € N@(n) under the 
lexicographic order < (see part (f) of Notation 1.1), with indices ranging in 
order from (0,...,0,7) through (n,0,...,0). In what follows, this description of 
¢(¢,v) will sometimes be convenient. 

We finish this section with a lemma that will be of importance later on. Define, 
for a € Nb, 8 € Ng and z € CP*4, 


(2.2) foa(z) = 228. 


We wish to compute the image of fag under the transform ®,jg)_|q)). In order 
to simplify our expressions, we first introduce some notation. 

Given y € M?*9(C) and 7 € {1,2,...,p}, let y;) denote the 7-th row of +. 
Let r(y) denote the element of C? obtained. by taking the sum of the rows of 
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y, and let c(y) € C? denote the sum of the columns of 7. Given a € Né and 
a € N§, let 1/?*4(a, &) denote the subset of M?*4(No) whose elements ¥ satisfy 
c(y) =a ands r(y) =a. 


Notice that M?™*4(a, &) is nonempty only when |a| = |@|. Now, ify € M?*4(No) 
and ¢ € D,,4, we may write ¢7 and 7! for 


Pp 
cy and [J yw)! 


= 


i=1 


i] 


respectively. Finally, when p and q are understood, we let M(a,@) stand for 
M?*4(q, &). 


2.5. LEMMA. Suppose that n > 0, a € N5 and B E N§ with |B| — |al =n. 
Then fag € FRY and 


@hoGod= YL eyes, 


nENE(n) yEM(a,B—n) 
nes 
where €¢ € Dp,g and v € C4. The order < is as in part (g) of Notation 1.1. 


Proor. Let ¢ denote ®, fag. According to Proposition 2.1, we have 


v) = L fop(Cw, w)e”“e-l"” dm(w) 


=f [ II(S-60e) leet eel dm(w). 


t=1 j=l 


At this point, we apply the generalized binomial theorem and obtain 


o(¢,v) = fe: [11 a Gppur SE) whet wert dm(w) 


SL ay €NG 
Iya) loa 
alc? x * +5 2 
= 5 — = wpe Mell" dm(w), 
Y: ca 
Yay Mp) ENG 
lyqayl=ou 


where ¥ is the p x q matrix with rows 7¥1),---,Y%p)- 

Now, we are able to compute the integral in the preceding line by expanding 
the exponential function into its Taylor series and then applying Lemma 1.2 
repeatedly. Doing so, we obtain 


1¢7 pF-7() BI 
(Gv= a (8, 7) eee 


yEMPX4(No) 
e(y)=a 


where «(8,7) = lif 8 > r(y), and «(8,7) = 0 if 6 — r(y) has any negative 
components. The result then follows immediately by gathering terms of the 
form 0"/n! for eachn € Né(n). 
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2.6. COROLLARY. Fizp=1,n>0 andy € N§ with |v| >n. Let fun € F}4 
be defined by 
pageaw ze 


and let d, = ®n(fin). Then 


yi (|v|—n)! .,_,\ 0 
bly= SD (Bene 


nENG(n) 
ngv 


3. An Inversion Formula 


The goal of this section is to give an inversion formula for the integral trans- 
form ®,, of section 2, when n > 0, p= 1, and q > 1. Throughout this section, p, 
q, and n will remain as in the previous sentence. We begin with the introduction 
of two mappings of O(B?, P(n, C?)) into itself. One mapping, P, is a projection 
that isolates a nonzero summand of ©, f,,,. The other mapping, L, is simply 
a weighted version of P which we then use to invert the transform ©,. The 
weighting of L is chosen to ensure the G-equivariance of our inversion formula. 

Let v € N@ with |v| > n and let fy.n(z) = 21”!""2%, as in Corollary 2.6. For 


each 7 € Né(n) we define a complex-valued function ws on B? so that 


(3.1) 6162) = FahenlGr) = Yo wT, 


nENG(n) 
as in (2.1). The functions ys were calculated in Corollary 2.6. 


3.1. LEMMA. Let v € N§ with |v| > n. Then there exists n € N§(n) such 
that yh” is not identically zero on B?. 


Proor. Since |v| > n, there exists 7 € N§(n) such that v > 7. By Corollary 
2.6, ym is the desired nonzero component. Mf 


3.2. LEMMA. Suppose that v,v’ € N§@ with |v'|,|v| > n. Furthermore, sup- 
pose that there exists n € N§(n) with 


V = wh? #£ 0. 
Thenv =v’. 


Proor. By Corollary 2.6, 7S” (¢) = C,¢’—” and ym (C) = C..¢""-", where 
CL,C, are nonzero constants. Hencev =v’. Wi 

Lemmas 3.1 and 3.2 show that f,,, can be recovered from a single nonzero 
summand y\”) (¢)v" /n!- of d.(¢,v) in (3.1). However, there may be more than 
one nonzero summand of ¢,. Our approach to the inversion problem is to first 
develop a method for isolating a nonzero summand of ¢,. 

We will construct a projection P that when applied to ¢, returns the n,-th 
summand y™)5™ /nv! in (3.1), where n, € N@(n) is maximal in the lexico- 
graphic order ~< with respect to ym? being nonzero. Restricting the w, action 
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of Definition 1.10 to K, we see that the K action on P(n,C%) (thought of as the 
fiber over ¢ = 0 of the holomorphic, homogeneous vector bundle E,, for G over 
G/K) is simply the natural action of U(q) by left translation. Notice that each 
v" is a weight vector for this representation, and that the lexicographic order ~< 
corresponds to the order on these weight vectors, from lowest to highest. Hence 
P will return the summand of ¢, of highest possible weight with respect to this 
action of K on P(n, C%). 


3.3. DEFINITION. For each j € {1,2,...,q} define Q; : O(B?,P(n, C%)) > 
O(B!, P(n, C4)) by 


Qi f(¢,v) = F (Cis oe $45 215.0) pares oe Gert) 


3.4. DEFINITION. If 7 € Né(n) has an immediate successor 7 with respect 
to the partial order <, let s(7) be the index at which 7 first disagrees with 7. If 
7 has no immediate successor, i.e., 7 = (n,0,...,0), let s(7) = 0. 


For example, 7 = (3,1,2,4,0) has immediate successor 7 = (3,1,3,3,0). So 
s(n) = 3, as 7 and 7 first disagree in the third component. 


3.5. DEFINITION. Suppose that 7 € Ng(n). Define P, : O(B%, P(n, C%)) > 
O(B9, P(n, C4)) by 


s(n) 
Py = II Q; 
j=l 
where s(7) is as in Definition 3.4. If 7 = (n,0,...,0), then P, := Id. 
3.6. DEFINITION. Define P : O(B?, P(n, C%)) + O(B?, P(n, C%)) by 
v0" Oo 
(Pé\(Gv)= So Poa pan (6%): 


nEN3(n) 


The role of the operator (v”/n!)(0"/0v") in Definition 3.6 is to single out the 
terms containing 0” in ¢, as in (2.1). Then P,, is applied only to those terms. 

For example, let q = 3 and n = 2. In this case the following table gives possible 
values of 7 € N@(2), in increasing lexicographic order, and the corresponding 
mappings P,. 


n s(n) Py 
(0, 0, 2) 2 Qi Qe 
(0, 1,1) 2 Q1Q2 
(0, 2, 0) 1 Qi 
(1,0, 1) 2 QiQe 
(1, 1,0) 1 Q1 
(2,0, 0) 0 Id 
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Any function ¢ € O(B?, P(2, C3)) is written using (2.1) as 
52 / ne 52 
8(6.9) SPO C) BO” Cuads Harpe OM (Ce, 
and the map P acts by P,, as given in the table, on the term containing 0”. 
Now let f € F,"* be given by f(z) = 23227324, so that v = (1,3, 1), in the 
notation of Corollary 2.6. The 7 € Ng(2) for which v > n are (0,1,1), (0, 2,0), 
(1,0,1), and (1,1,0). It follows from Corollary 2.6 that 
~2 
Vv aes eeres 
(S2f)(C,v) = 180, G3 02%3 + 36¢1C2¢3-5¢ + 6630103 + 18¢3C301 02. 


Using Definition 3.6 and the table above, we compute that for ¢ = ®2f, 


PO(C,v) = 18C3G301d2. 


Notice that only one term of ¢(¢,v) remains, the term indexed by the maximal 
n € N3(2) with respect to lexicographic order for which ~™ £0. This is no 
coincidence, as is shown by the next lemma. 


3.7. LEMMA. Fit v € N8 with |v] > n. Let fun(z) = 2i’'-"2Z%, and let 
Cio eneNS(n) ho" /n, as in (3.1). Let a € N§(n) be mazimal in 
the order < with respect to yo being nonzero. Then Py (52 /al = prae/al 
and for n <a, P, wh”? 07 /n! =0. 


PRoor. Since ¥)\” is a monomial in ¢, then either P, wo /n! = yh” or /n! 
or Pw? 57 /n! = 0 for any n € Né(n). We first show that Pw o%/al = 
yar /al. 

Suppose that a = (n,0,...,0). Then by Definition 3.5, P, = Id, so we have 
Pas o®/al = yh o*/al trivially. 

So we now assume that a = (a1,...,a@) # (n,0,...,0). Let @ be the im- 
mediate successor of a within N(n). Note that @; = a; for 1 <7 < s(a). 
Also, Gs(a) > Ys(a), and |@| = |a| =n. It follows that there must be a nonzero 
component a, of a with s(a) <t <q. Also, by Corollary 2.6, we have that 


VC) = Cay... Gas, 


where v > a and C, is a constant. For a contradiction, assume that Pwr /a!l 
= 0. Then, by Definition 3.5, there exists k € {1,...,s(a)} with y, — a, > 0. 
Consider a! = a — e + €;. We have that a’ € N§(n) with a’ > a. Furthermore, 
by Corollary 2.6, ys? #0. This contradicts the maximality of a. 

Fix 7 < a. The proof will be concluded by showing that Pron /ni = 0. 
If yp = 0 then the result is immediate. So, for a contradiction, assume that 
wv £0 and that P,b'o /n! = y$?97/n!. Definition 3.5 and Corollary 2.6 
then imply that v; — n; = 0 for € {1,...,s(n)}. Now, since a > n, there exists 
k € {1,...,8(m)} such that a, > m,. However, then 4%, — a, < 0, and so by 
Corollary 2.6, u§* = 0. This contradicts the fact that y{*) 40. 
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We now construct another mapping of O(B?, P(n, C2)) into itself that we will 
use to weight the mapping P. Essentially, we will be assigning to each element 
¢ € O(B?, P(n, C?)) a solution of a certain partial differential equation. 


3.8. DEFINITION. Suppose f € O(B?, P(n, C?)), ¢ € B%, and v € CY. Fora 
given non-negative integer m and an index j € {1,...,q}, we define the mapping 
I(m, j) : O(B4, P(n, C?)) > O(B4, P(n, C2) by 


(I(m, j)f)(¢, v) 
tm te 
-[[- Fas Gas tj Gots +1 Gqr 0) dba da... dtm; 


with the convention that eee v) = f(¢,v). 

k! 
(k+m)! 
3.9. DEFINITION. Define F,, : O(B?, P(n, C2)) + O(B?, P(n, C?)) by 


For example, if f(¢, v) = Cfet, then (I(m,7)f)(¢,v) = CF of. 


(Fy f)(G,») = (1(m1)1(2,2) --- Ls DF) (G%), 
where 7 € Né(n). 
Suppose that f € O(B%, P(n, C%)) and 
= 35 Dae 
nEN§(n) YENG 


is the Taylor series expansion for f centered at the origin. Since the series above 
converges uniformly on compact subsets of B?, it follows from Definition 3.8 that 
if p € NG(n), 


Fof(4,v) 


Dae S On, Fp(C70") 


neNg(n) yEN3 


(3.2) = ye s eee C757. 


nEN§(n) yENG 





Now, we weight the mapping P by the F;, to obtain a new mapping that will 
play a large role in the rest of the paper. 


3.10. DEFINITION. Define L : O(B?, P(n,C?)) + O(B?, P(n, C%)) by 


ao 
(L6)(x) = DS) FrPpv" 55 (62)- 
neEN3(n) 
For example, recall that if f(z) = 272.2324, and ¢ = ®2f, then 


P¢(¢,v) = 1863 C301 %2. 
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Applying (3.2) and Definition 3.10, we compute that 


LAG, 0) = Brey Eat = BCR v 


((0, 2,1) + (1,1,0))! 


For ¢ € O(B?, P(n, C2)), we express ¢ as the sum 


or) = TT wOS 


NENG (nr) 


as in (2.1). For a compact subset A of B?, set 


(3.3) Ida = SS WK, 


nENG(n) 


where |- |, is the supremum norm over A. We endow O(B?, P(n, C?)) with the 
topology of uniform convergence on compact subsets of B, i.e., the topology 
induced by the norm of (3.3). 


3.11. Lemma. The mapping L : O(B%,P(n,C%)) — O(B%,P(n, C%)) is 
continuous. 


PRooF. The remarks after Definition 3.9 serve to show that F, is continuous 
for each 7 € N§(n). It remains to show that P, is continuous. 

Let ¢ € O(B%,P(n, C%)). Suppose that {¢,} is a sequence of functions in 
O(B?, P(n, C%)) such that {¢,} converges to ¢ uniformly on compact subsets of 
B’, in the norm of (3.3). Let A be compact in B? and for 1 < j < q, let A; 
be a compact subset of B? containing the natural projection of A onto the set 
{¢ € B4 | ¢; = 0}. Then 


10; — Q;¢ella < |lo — della;- 


It follows that Q; is continuous and hence FP, is continuous. Wi 
We are now ready to invert the transform ©®,, of section 2. 


3.12. THEOREM. Suppose that ¢ € ®,(F}%), and that d6 = @,f, for f € 
F149, Then, for z € C7}, 


f(z) = Jim Fe [, (Ld) (16, v)e(+e7)#5 ell” dmv jamie). 


PRooF. The theorem is proven in two steps. First, given vy € N§ with |v| > n, 
we let fin(z) = 2) "Z% and we let ¢, = ®nfvn. We will prove that the 
theorem holds for ¢,. Then we will show that the theorem holds for an arbitrary 


$ € &,(F} 4). 
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So, let v be as above, and let n, € Né(n) be such that P,, (wh a /m!) #0. 
Then by Lemma 3.7 and Definition 3.10, 





(LONG) = DP, ZH (0) 


neNg(n) 
v\(jv| — n)! 
= F,, Pr, CY ge 
NW * 1 (v— nv)! 
(3.4) = (|p| —n)ICe-™ 5, 


Hence, by using (3.4), (1.3), and Lemma 1.9, we compute that 


re [. [cone vjelaste 2s ell” dm(v)dm(C) 


= t"l-"+4(\p] — n)! [ CY Bm (6+ 2s ell” dm(v)dm(C) 
Bs Jc 
q 

=ientezie (my! [com exp|n (SFG) lem 

Be j=l 

lul—noy 
— zlul—nt+@ (lv —n)! U—Ny Pv—ny 
pa [ermeemdm(g 

= tl’ 








(lv| —n +4)! 


are pee = mt 


and so 


lim ale f ie (L¢,)(t¢, yeasty” zg eT lel? dm(v hat) 


to1- dt@ 
a (FE tense [2 Oe] — 2)! 
= tim G5" al (= n+) ) 
= fiun{z). 


This completes the first part of the proof. 
Now let ¢ be an arbitrary element of ®,(F17), 6= @,f for f € Fi-4. Put 


f(z) = S- ay frn(2). 


veNng 
[vl>n 





This series converges uniformly on compact subsets of C?t+! and in the norm on 
F149 induced by (-,-). Hence by Lemma 2.3, given ¢ € B?, 


(3.5) o(¢, v) _ SS ay dr (C, v). 


veEeN? 
[vlan 


Moreover, the series in (3.5) constitutes a Taylor series expansion for ¢, and so 
the series converges uniformly on compact subsets of B?. Hence by (1.3) and 
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Lemma 3.11, for ¢ € [0,1) we have 


eee» 


tur) (tC, v)e(aS te" )2s enh? dm(v)dm(C) 


oe 
=f. is sy a,(Ld,)(t¢, eS") el” dm(v)dm(C) 
2 jan 
q —iemmentsimee 
(3.6) =f S- tl¥l—"q, (\v} — n)!c’- ae exp|a Ds zat) dm(¢). 
vend j=l 
ll>n 


Notice that we may exchange integration over C% with summation in the 
equation preceding (3.6) due to the absolute convergence of the integral in v. 
We finish by showing that we may interchange integration and summation in 
the right hand side of (3.6). The theorem will then follow immediately from the 
first part of the proof. Thus by the Fubini-Tonelli theorem, it suffices to show 
that 


q 
cue Zz exp E OS 7n6)| | dm(C) < oo. 


j=l 


SS lav|(e| myer [ 


vend 
Ivjan 





Note that for a fixed z € C?t+!, the function 


Ce ae exp E ( znG) | 


j=l 


is bounded in modulus on B?. Hence there is a positive constant c, such that 





F lave meen fi forom ee exp[es (S278) fem) 





vené j=l 
[vl>n 
(3.7) Sez SD Jar|(v|— nye | -™ |dm(0). 
vend Ba 
[el>n 


Now by applying the Cauchy-Schwarz inequality to the integral in (3.7) and 
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using Lemma 1.9 we compute that 


cS faul(lo|— nyse fF erm dic) 





ape 
; vicn( (V-m)! \3 
<¢, p> ae le| — nytt ( ) 
lv[2n 
: ; 2(|v|—n) 4 
<4, 3. loll o0! (Tae) 
ae 
(3.8) 


aL 


. aps pelle ni | » a= aot nt 7 : 


veng vené 
lvy[2>n |vlzn 





By (1.2), the first factor of (3.8) is simply (f, f \2, while the second factor is 
finite by Lemma 1.4. Hence the product in (3.8) is finite, and so the proof is 
concluded. @ 


4. A Hilbert Space Structure for ®,(F}) 


Recalling Theorems 1.7 and 2.2, observe that for n > 0 the space ®,(F)7) 
is an irreducible subspace of O(B%,P(n,C%)) under the representation wn of 
U(1,q). Let J, denote this subrepresentation. One would like to explicitly 
describe a Hilbert space structure on ©,(F)7) with the property that J, and 
On Will be unitarily equivalent via ®,. It is worth pointing out that co, is not in 
the discrete series for any n > 0 ifp = 1 and q> 1. If p=q = 1 then the ladder 
representation of spin n > 1 is in the discrete series and the representation with 
spin n = 0 is a limit of discrete series (see Theorem 4.4 of [3]). 


4.1. DEFINITION. For $1, ¢2 € O(B?, P(n, C?)), define 
($1,¢2)ng 2 = Jim aa [ ; ie (L$1) (#6, v) ba(6, 0) eh" arn(vdm(¢)| 
= lim mle f ((Lo1) (tC, -), d2(¢, -)) an(0)) 


t~17 
whenever the integral above converges. Here L is as in Definition 3.10. 


For ¢ € O(B%, P(n, C*)), put ||4||?,, := (¢,¢))n,q- It is easy to show that on 
the set {¢ € O(B?, P(n, C4)) | ||d||2,q < co}, the form ((-,-))n,q is hermitian and 
positive semi-definite. We next examine its behavior on ®,,(F,°%). 


4.2. THEOREM. Let ¢,,¢2 € ®n(F}%) with d; = Onfi and $2 = Onfo for 
fi, fa € FR, Then 
(¢1, $2) nq = (fis fa). 
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Hence ((-,-))n,q defines a positive definite hermitian form on ®,(F})9). 


Proor. Put 
filz)= S2 afin(z) and fo(z)= S> br fin(z) 
vENg veNé 
|v[>n [v[>n 


where f,,, is as in Corollary 2.6. Correspondingly, according to Lemma 2.3, we 
have 


bi(6,v) = D> ad.(6,v) and do(¢,v) = S$) b4.(¢,») 


q q 
vENg ven} 
[vl2n lular 


where ¢, = ®,f,,,. Then by using the series expansions above, along with 
Lemma 3.11, Corollary 2.6, (3.4), and Lemma 1.2, for t € [0, 1), 
J, [ eaee, baw} amo)am(c) 
Ba 


= i dS abs (Ler) (t6, v) bo v) e7!" dm(v)dm(¢) 


ven veNné 
Iel>n |el>n 
h vien (¥] — 2){{(|a| — n)!o! v—n, xB- 
(4.1) =f | x a,bst!! Seance ata m (PM | dm(C). 
v,vEN 
lie 


Since t € [0,1), by the dominated convergence theorem we may interchange 
integration and summation in (4.1). After doing so and applying Lemma 1.9, 
the only nonzero terms remaining in the sum are those for which v = 7, yielding 
that 


i S, a,bgtei-n Mu = n)\(121 _ semen] dm(¢) 








v,DENg (o—m)! 
ae 
vy! 
— luyl—nii) — : 
= 2 aha are 
lp[>n 
a,b, t”!-"vl(|v| — n)! 
7 -2. (ul —n+q)(y}—n+@-))...(e—n+]) 
ll>n 


Hence 
sim ele, [ (ceantec. yaa) e-*" ent wdc] 
= > a,b, (\v| — nu! 


vené 
|y[2n 


=(fi, fo). a 
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4.3. COROLLARY. The vector space ®,(F}1), when endowed with the inner 
product ((-,-))n,q, 1s @ Hilbert space. 


PrRooF. By Theorem 4.2, ®,(F}:7) is a pre-Hilbert space. So all we need to 
show is that ©,,(F}'7) is complete. Suppose that {¢;} is a sequence in ®,(F}'7) 
that is Cauchy in the norm induced by ((-,-))n,¢, and suppose that ¢, = ®n(fx) 
for f, € F147. Then by Theorem 4.2, the sequence { f,} is Cauchy in the Hilbert 
Space F}7. Hence there is a function f € F}-% such that f, converges to f in 
norm. Let 6 = @,f. Then ¢ € &,(F/'7) and ¢, converges to ¢ in norm. Mf 


4.4. COROLLARY. The representation 0, 1s irreducible and unitary on the 
Hilbert space ®,(F}:7) and is unitarily equivalent to o, via ®n. 


PROOF. This follows from Theorems 2.2 and 4.2. Mf 

At this point one may wonder whether the inner product ((-, -))n,¢ characterizes 
©,,(F,77) . In general, this inner product is not enough to characterize ®,,(F}:7). 
However, when we put the inner product together with the differential equations 
of Theorem 2.4, we are able to combine our results with those of [4] to obtain a 
complete characterization of ®,,(F} 7). 

For q > 1, let Di,, denote the set of differential operators consisting of all 
2 x 2-minors of the matrix of differential operators 


a Se on 
O.,= ( <i > 
Bo 
of Theorem 2.4. If g = 1, we will say that D, ; = @. Also, let 
Xp = {6 € O(BY, P(n, C4) | ||¢||2,. < co and D¢ = 0 for all D € Dy 4}. 


4.5. THEOREM. Letn € No andqéN. Then ,(F}%) =X}. 


Proor. From Theorems 2.4 and 4.2, we know that ®,(F)7) c ¥}4. So it 
remains to verify the reverse inclusion. 
Suppose that ¢ € ¥}4. Then 


$(¢,v) = > d(m)( ¢,v) 


where each ¢(m) is a P(n, C%)-valued holomorphic polynomial that is homoge- 
neous of degree m on B?. Since 0 = D¢(¢,v) = 3-9 Detm)(¢,v), and since 
the operator D preserves the homogeneity, hence the linear independence of the 
functions D¢im), we conclude that D¢(m) = 0 for all m € No. The proof of The- 
orem 4.2 implies that any polynomial in ¢ has finite norm. Hence $(m) € Pea 
for each m. 

In the case that d(n,q) > 1, the results of [4] imply that each ¢/m) is also an 
element of @,,(F}7). In fact, this also holds in the line bundle case, i.e., when 
d(n,q) = 1. Corollary 2.6 shows that when d(n, q) = 1, the image of a monomial 
is a monomial. Thus in this case we can invert ©, directly from Corollary 2.6. 


m 
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So, with $(m) as above, there exists fim) € Fi4 with ®,, fim) = %(m) for each 
€ No. By Corollary 2.6, f(m) must have the form 


fim) (z) = S- Om, v2 IWP Ze, 


vené 
[vl=mtn 


Put f=, f(m)- Since oe ¢(m) converges to ¢ uniformly on compact subsets 


of 


B? (see (3.3)), we may apply the dominated convergence theorem as in 4.2, 


along with Corollary 4.3, to get 


IlPlln,4 


N 
: 2 

nim, I D. bemllng 

N N 
wm (D2 Fem)» Do Fem) 

N 

lim S* So Jam,v|?(\v] — n)tu! 
NO vend 


|ul=m+n 
= (f, f). 


Hence (f, f) < oo, and so f € F149. From the continuity of ®, we then conclude 
that ®,f=¢ 


10. 


11. 


12. 


13. 
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FOURIER INVERSION IN 
SEVERAL VARIABLES — A NEW 
LOOK AT AN OLD PROBLEM 


MARK A. PINSKY 
Dedicated to Ray A. Kunze 


ABSTRACT. We describe some results which give necessary and sufficient 
conditions for the convergence at a preassigned point of the spherical partial 
sums (resp. integrals) of the Fourier series (resp. Fourier integral) in the 
class of piecewise smooth functions on Euclidean space. These are also 
formulated for orthogonal series expansions in the eigenfunctions of the 
Laplace operator of a rotationally invariant Riemannian manifold. The 
conditions are vacuous in one and two dimensions, but yield non-trivial 
examples of divergent Fourier series/integrals in higher dimensions, with 
respect to a preassigned point. 


1. Introduction 


The problem of pointwise Fourier inversion in several variables is more than 
fifty years old ({B1], [B2]). For periodic piecewise smooth functions on the real 
line, the simplest results of classical Fourier analysis imply that the partial sums 
converge at every point to the normalized value of the function. The same result 
holds for periodic piecewise smooth functions on the plane. When we pass to 
three dimensions or higher, this is false. We show below that, for the indicator 
function of a three-dimensional ball, the spherical partial sums computed at the 
center of the ball oscillate between two positive limits. This also holds for: a) 
the Fourier integral expansion with respect to all of R°; b) the Fourier sine series 
with zero boundary values on any containing cube; c) the radial Fourier-Bessel 
series with respect to any containing ball, d) the spherical harmonic expansion 
(on the surface of S*) and e) the hyperbolic Fourier transform on the hyperbolic 
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space H®. If we consider the difference of two such functions we obtain a simple 
example of non-localization in three dimensions: the indicator function of an 
annular region is zero in a neighborhood of the origin, but the Fourier expansion 
diverges at the origin. 

For more general piecewise smooth functions we find a condition for conver- 
gence at a pre-assigned point. This condition is necessary and sufficient for the 
Fourier integral expansion with respect to all of R”, as well as for the corre- 
sponding expansions in spherical harmonics or the hyperbolic space. In the case 
of Fourier series we are limited to the discussion of radial functions, where we ob- 
tain a necessary and sufficient for convergence at the center. The Fourier-Bessel 
expansion is generalized to eigenfunction expansions on a class of rotationally 
invariant Riemannian manifolds. 

Our viewpoint stems from the idea that Fourier analysis is a particular ex- 
pression of the spectral theory of the Laplace operator. In particular we always 
use spherical summation as a definition of convergence. It is an open question 
to determine if the above results are valid for other definitions of convergence. 


2. Convergence of Fourier transforms 


A function f € L1(R") is piecewise smooth with respect to x € R” if there 
exists a finite set 0 = ro(x) < r(x) <...<rx(x) so that the mean value 


1 


Wn-1 





es fz (r) = 


q, f(x+rw)dw 
gn-1 


is smooth on each subinterval (r;_1,7;) with right and left limits f,(r; +0) and 
is identically zero for r > r(x). 
The smoothness index of f at x is the integer defined by 


j(f;x) = max{j:r — f,(r) is differentiable of class C”}. 


In general we have j(f;x) > —1 for any piecewise smooth function. j(f;r) =0 
signifies continuity with a discontinuous derivative. The condition j(f;x) = +1 
signifies a continuous first derivative only, etc. etc. 

The Fourier transform and spherical partial sum of an integrable function are 


defined b 
efined by fw) = 1 if e 42) F(x) dx 
(27)" Jpn 


fu(x) = / Peele 





If f has n derivatives in L' then it is a classical fact that limatoo fu (x) = f(a). 
For completeness we give a quick proof in the appendix. If f has less smoothness, 
then we have the following result, which describes the convergence of the Fourier 
transform in terms of the smoothness index. 


THEOREM 1. Suppose that f is piecewise smooth with respect to x € R”. 
Ifn =1,2 then limmjoo fm (2) = fr(0 + 0). 
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If n > 3 then limmjoo fu (2) exists iff r — f,(r) has [252] continuous deriva- 
tives, i.e. j(f;x) > [(n—3)/2]; in this case limmtoo fu (x) = f,(0+0). Otherwise 


we have 


—oo < liminf M~”[fu(x) — f-(0+ 0)] < limsup M~" [fy (x) — f.(0 + 0)] < 00 
Mfoo Moo 


where v = 252 — j(f;x) and j(f;x) is the smoothness index defined above. 


Many authors have proved equiconvergence theorems which reduce a multi- 
dimensional Fourier expansion to a corresponding problem of one-dimensional 
Fourier analysis; see [CCTV] for some recent results in this direction. The above 
theorem is of a different character, dealing specifically with the dimensionality 
in the statement of the conditions for convergence. The next section clarifies the 
situation in terms of a canonical example. 


2a. An example in three dimensions. 
Before turning to the proof of theorem 1, we give a basic example in which 
all of the computations can be made explicit. This is the indicator function of a 


ball in R?: 
1 if0<|z|<a 


0 otherwise , 


pea) ={ 


where a is any positive real number. The Fourier transform is 


fa 1 j —i(y,2) 
=, CR aa Ho 
f (u) (27)3 ice 


For ps = 0 we have f*(y) = Oxy x 4na°, whereas for 4 # 0 we have 


1 oe 
Pu) = sara sin rin) rar 





a ot eral Sint, 
an? |p? |u| 
= F*(|u)). 


At the center of the ball, the spherical partial sum is 


f%,(0) = / en Pe 





M 

= i F%(y)4my? dy 
0 
of csi 

= a (2 — acosay) dy. 
T JO y 


The first term is a convergent improper integral and the second integral is ex- 
plicitly evaluated. Thus we have 


fi(0) = 1- = sin Ma + 0(1), M — oo. 
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In particular, we have the exact fluctuation of the partial sums at the center: 


2 2 
liminf f%,(0) =1— = 2(0)=1+— 
im inf fx, (0) a inysup fu(0)=1+— 


By analogy with elementary Fourier series, we may define the Gibbs constant as 
the difference between the upper and lower limits, normalized by the disconti- 
nuity of the function at the edge: 


G = limsup f§,(0) — lim inf fx, (0) = : = 1.2732... 


M-—0o 


In contrast with the ordinary Gibbs phenomenon of one-dimensional Fourier 
series, here we have the additional fact that the partial sums diverge at a point. 
This is impossible for the one-dimensional Fourier series of a piecewise smooth 
function. 

The above example can be modified to demonstrate the failure of localization 
in three-dimensional Fourier analysis. Specifically, let 0 < a < 6b and let the 
indicator function of the annulus be defined by f%° = f° — f*. Following the 
computations above, we have 


2 
26(0) = ; (sin Ma — sin Mb) + o(1), M => 
so that the spherical partial sums at z = 0 have no limit when M — oo. But 
the function f%° is zero in a neighborhood of the origin. 


If one carries out the above computation in higher dimensions, one finds (in 
accordance with theorem 1), that the spherical partial sums satisfy 


—oo < liminf M°=" fx, (0) < limsup M°=" f2,(0) < 00 
M-—+oo M-—oo 


2b. Proof of Theorem 1. 
The proof depends on the integral representation of the spherical partial sum 


fi Gees [ " Fe(r) DM (rn) o®) dr 


where the Dirichlet kernel D (r) is defined as the integral of the complex expo- 
nentials over a ball: 


1 : 
M = 1(u,z) 
Da (lal) (2n)" Fai ne 


where we have specifically noted the radial dependence. This may be expressed 
in terms of a Bessel function by the formula 


Jnj2(Mr) 


M = Mn 
Dy (r) =C (Mr)r73 
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where the constant depends only on the dimension n. Using known properties 
of the Bessel functions, we have the following fundamental properties of the 
Dirichlet kernel [SW]: 


DM(0)=C,M", M-> 00 
;Ra1 


Ma 
DM (r) = Cage [sin(Mr ~8,) + O(—2)), r>0,M— oo 
| pa’ ae 


1d 
Dii(r) = —s— Da alr), n>3,r>0. 


Using this, we can write 


fu (2) = Wn | * Falr)DM (rye! dr 


Wy-1 


a cay d 
ie is * flr) = Dnt alr) dr. 





We break up the integration on each sub-interval (r;_1,7:) and integrate by 
parts; thus 


[ork Zia) ar 
=f (DM airy = [Ele Fal DM alr) ar. 


When we sum over i, the boundary terms simplify to the jumps of the mean 
value function, while the new integral corresponds to a lower-dimensional Fourier 
partial sum: defining éf,(r) = f2(r +0) — f2(r — 0), we have in detail 


Qn if fr(r)r”—! D™ (r)dr 
0 
pee ~ of. n—2)M “d n—2 fF D™M d 
= LshelosyreDitalos) + fe Fel DA a) der 


If 6f,(ri) # 0 for some i, then the first term is non-zero and, from the asymptotic 
behavior of the Dirichlet kernel, dominates the asymptotic behavior of the second 
integral when M — oo. If, on the other hand, all of these jump terms are zero, 
then we may reduce to the consideration of the lower dimensional integrand. In 
particular if nm = 2m +1 is odd and j(f,x) > m—1 = [25%], then 6f,(ri) = 0 
together with all derivative terms of orders less than or equal to m— 1. By 
repeated integration-by-parts m times we arrive at a one-dimensional Fourier 
integral. By one-dimensional Fourier analysis this has a limit, which can be 
computed by inserting a smooth function. From the classical theory (see the 
proof in the appendix), this limit exists and equals f,(0 +0). If, on the other 
hand, n = 2m +2 is even and j(f;z) > m—1 = [25%] then éf,(r:) = 0 
together with all derivative terms of orders less than or equal to m— 1. By 
repeated integration-by-parts m times we arrive at a two-dimensional Fourier- 
Bessel integral for a radial function, for which the convergence is proved directly. 
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3. Convergence of Fourier series of periodic functions 


Passing to the analogous case of Fourier series, we first examine some concrete 
examples in R°. 


3a. Examples in three variables. 

In this sub-section we consider the Fourier series of the radial function f : 
R? — R! defined by 

; 1 if0<|z|<a 
f(z) = 
0 otherwise. 

Here 0 < a < 7, so that we are looking at the indicator function of a ball which 
is inside (possibly touching the edges of) the basic cube. 

The Fourier coefficients are computed easily by evaluating the integrals in 
spherical coordinates r,¢,0 as follows: 


1 : 
An = (—y* i. e() dri drodr3 
an” Jixi<a 


1 a “ie v . 
= ey | [ i e Inlr cos $n? sin g dO dd dr. 
0 0 —T 


For |n| = 0 the integral is easily seen to be Ao = ot For |n| 4 0 we may 


evaluate the integral by elementary calculus to obtain 


1 mas 
Ay = ama sin(|n|r) r dr 


ae! gates |nja sin Inja 
~ 2? |n| (n| In|?“ 
The partial sums of this series for a = 7, |n| < 134 and (x, £2, 43) = (z,0,0) 
have been graphed using Mathematica and are depicted in Figure 1. The partial 
sums for |n| < 155 are depicted in Figure 2. 








pe 











Figure 1 Figure 2 
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We note a dramatic oscillation in the neighborhood of x = 0, reminiscent of 
the Gibbs phenomenon from one-dimensional Fourier series. But in this case 
we have divergence of the partial sums by oscillation, which is totally unknown 
for one-dimensional Fourier series. The oscillation displayed by Figures 1 and 2 
illustrates that some of the partial sums go above the line y = 1 while others 
go below. We prove in section 3b that the Fourier series diverges at x = 0. 
However we have not been able to compute the Gibbs constant lim sup, fr (0) — 
lim inf yy fu(0). 

This example may be modified to produce a simple example of the failure 
of localization in higher dimensions. We consider the indicator function of an 
annulus: 

f(x) =1 for a < |x| < 6 and f(x) = 0 otherwise. 


Here 0 < a <6 <7. The function is zero in a neighborhood of x = 0 but the 
Fourier series diverges by oscillation at x = 0. Figures 3 and 4 are graphs of 
the partial sums of this series with (x1, 72,13) = (x, 0,0). In Figure 3, a = 7/4, 
b = 7 and |n| < 134 ; in Figure 4, a = 1, b = 3 and |n| < V/125. We see a 
phenomenon similar to the one displayed in Figures 1 and 2; some of the partial 
sums are positive at x = 0 and others are negative. 








Figure 3 Figure 4 


Previous examples of the failure of localization in low dimensions have de- 
pended on limiting procedures, such as infinite series, the principle of uniform 
boundedness or other non-constructive methods of functional analysis. We refer 
to the article of Ash [A] and the references cited therein. 


3b. Proof of convergence/divergence. 

The n-dimensional torus is T” = {x = (a1,...,%n): —™ < a; < a}. The 
Fourier coefficients and spherical partial sums of an integrable function f(x), 
x €T” are 


1 : 
Am = —— (x)e—*™) da, m€ Z”" 
(27)” Tr ) 
a= DS. Anes xz €R” 


lm|<M 
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The spherical mean value is 


fer) = —— | set rudd 


Wn—-1 





where we extend the function f periodically to R”. 


DEFINITION. f € L'(T™) is a piecewise smooth radial function with respect 
tox € T” ifr — f,(r) is piecewise C° and the periodic extension satisfies 
f(cwt+rw) = fp(r),0<r<m,weES"™! and f(x+rw) =O0ifr>7,rweET. 


In two dimensions the proof of convergence for a piecewise smooth radial 
function can be accomplished by appealing to a theorem of Hardy and Landau 
[HL]. Otherwise we proceed as follows; for a radial function the Fourier coefficient 
is the restriction of a smooth radial function: A, = A(|m]) where the asymptotic 
behavior of A(r), A’(r),r T 00 is determined explicitly. The spherical partial sum 
is expressed as a Stieltjes integral: 


M 
fu():= >> Am = f A(r) dN(r) 


|mj<M 
where N(r) is the counting function of lattice points, satisfying Landau’s estimate 


N(r) := - 1 =c,r" + O(r"~2+ wat) Tr — oo. 


|m|<r 


This is used to write 
fu (0) - s A(r)en nr”! dr 
: M 
= A(M)[N(M) — ¢mM™] + i A'(r)[N(r) — car} dr 


and leads to the following theorem, which extends the results in [PST]. 


THEOREM 2. Suppose that f is piecewise smooth and radial with respect to 
xéeT”. Ifn=1,2 the Fourier series converges, and 


im, fu(e) = fe(0 +0). 


If n > 3 the Fourier series converges iff the function r — f,,(r) has at least [252] 
continuous derivatives. 


3c. Fourier series with boundary conditions. 
The methods of the preceding section can be used to study the expansion of 
a radial function in an eigenfunction series in the cube —7 < 2; <7: 


f(z)~ S- An¢m(Z) 


mez” 
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where the eigenfunctions ¢,, satisfy Ad + Ad = 0 in the cube and the Dirichlet 
condition that ¢ = 0 on the boundary. (The corresponding problem with Neu- 
mann or mixed Dirichlet-Neumann boundary conditions can be dealt with by 
suitably modifications) 

Separation-of-variables produces the eigenfunctions in the form 





MZ; 
omi,... ee (Bs Bee Yeu) = Ii. 1 (cos 








TET ai 60s ua) 
sin —— + si 
2 2 2 


The Fourier coefficients of a radial function f(x) = F(|z|) are computed as 


foro ae il F(|z|) Dive aie Ely oe fn 
|x| <a 


At the center of the cube we have 


d, Ambm(0) = An Tn sin 


m; odd 
But for m; odd, 


mt 
sin —— dx 
2 





Am =a f F(|z|) H"_, cos = 
|z| <7 


so that 





S> Am¢m(0) = my F(|z|)TI_, cos = sin? i" ae. 


To complete the analysis of the Fourier expansion, we need the asymptotic 
behavior of the number of positive lattice points, defined as 


NPOoe= yo 


n 
mi>0,|m|<A 


PROPOSITION. When \ — 00, N+(A) = end" — fadX"~! + O(A" 2 +47), for 
suitable positive constants en, fr. 
A corresponding analysis leads to the following identical result: 


THEOREM 3. Suppose that f(x) = F(|z|) where F is piecewise smooth on 
[0,7] and zero elsewhere. Then the series expansion >>, Amom(0) converges iff 
F is of class [253]. 


4. Spherical harmonic expansions 


The expansion of an L? function on the sphere S” is written 


pr So¥ 


m=0 
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where yo Je Hm, the space of spherical harmonics of degree m with respect to 
the sphere S”. The partial sum fy := SCoc ney yo is the orthogonal projec- 
tion of f onto the linear subspace of spherical harmonics of degree < M. The 
spherical harmonics of degree m are SO(n + 1)-invariantly defined as homoge- 
neous polynomials of degree m which satisfy Laplace’s equation Agn+iY = 0. 
In order to compute fr(Q), we may take a basis of spherical harmonics with 
respect to the axis Q € S”. Without loss of generality we will suppose that 
Q = (1,0,... ,0) is a fixed origin from which distance on the sphere will be mea- 
sured. A system of geodesic polar coordinates is written (r,6) , whereO<r<a 
and 6 € S"~!. With this notation the metric of S” is written 


ds* = dr? + sin’ r dé? 


and the Laplacian is written 


: +(n—1) cot é + iA 
—> + (n- 1) cotr— + —,—Agn- 
dr? dr sintr > 
An n-dimensional spherical harmonic is written 


YA (r, 8) = o(r)4(8) 


where the radial factor ¢ is a solution of the ordinary differential equation 


Asn = 


6” +(n—-1)cotr¢’ + ( 





A = |m|(|m|+n—1) 


and w(@) is a spherical harmonic on S”~!, solution of 
Agn-1p + By = 0. 


An integrable function f(Q), Q € S” is said to be piecewise smooth if for all 
Qo € S” the spherical mean value 


(0,7) ar fa (r) = J 


ds 
Wn—1 Te deaen hed 


is piecewise C® on the interval 0 < r <7. The Fourier expansion of a piecewise 
smooth function is written 


f~ >> Ak.mbk,m()bm(8) 
k,m 





where 
A me Poe So. £(7, 0) bk,m(7)Wm(9) sin”? r dr dw 
fe Jpn Fe Gm (7)2bm(0)2 sin"! rdrdw 
From the above remarks, we have the series at r = 0 in the form 


0) ~ S- Akx,odk,o(0) 


k>1 





and the f-dependence of the Fourier coefficient is given by 


[ in f(r, 8) bk0(r) sin”! r dr dw = wp_1 a fo(r) bx,0(r) sin”! r dr. 
Sei Jp 0 


This shows that the series for f(0) depends only on the spherical mean value 
fo(r),0 <r < m. Therefore we may restrict attention to radial functions and 
obtain the following result. 
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THEOREM 4. Let f(Q),Q € S” be a piecewise smooth function. Then the 
spherical harmonic expansion converges at Qo € S” iff the spherical mean r — 


fo,(r) has (2=3] continuous derivatives. 
Qo 2 


5. Dirichlet eigenfunction expansions 


In the previous sections we presented results on the convergence of series 
expansions of functions on complete manifolds without boundary, where we have 
the full group of translations, allowing any point as the origin. When we pass to 
manifolds with boundary, we have a privileged origin, where we can study the 
convergence/divergence of the expansions. To motivate the exposition, we first 
present the case of a Euclidean ball. with Dirichlet boundary conditions, where 
the analysis is explicitly described by Bessel functions. Previous studies of this 
case with graphical illustrations are contained in [GP] 


5a. Analysis on R”: Fourier-Bessel series. 
We consider the Euclidean space R” with a ball of radius a > 0: 
B, = {x :0< |z| < a} 


The n-dimensional Laplace operator is A = >”_, 2. 


Let ¢ = ¢k,m(x) denote the eigenfunctions of the n-dimensional Laplace op- 
erator in the ball: 
Ad + A¢ = 0, 0< |x| <a 
g(x) =0, |x| =a. 
These may be computed in terms of Bessel functions and spherical harmonics 
by the technique of separation of variables according to 


Jimaied(n—2) (PV) (m,n) 
m| “ |m|+3(n—2) cs azo = Ly, 
pim|+3(n—2) Ym(w), r= |x|,w=x/|x|, A eo 


$k,m (x) — r 





Here the first factor denotes the standard Bessel function and the second factor 
is a spherical harmonic of degree |m|, a solution of the equation Agn-1Y + 
|m|(|m| + — 2)¥ = 0, where Agn-1 is the spherical Laplacian operator. The 
quantity x") denotes the kt zero of the Bessel function J\m|+1(n—2): The 
index k = 1,2,... while the index m takes values in a finite set which labels the 
spherical harmonics of degree |m| = 0,1,2.... These functions are orthogonal 
with respect to the Euclidean volume element in the ball B,, which, when written 
in polar coordinates is 


Ce ee i bres xing (®) Pha ana (0) EX 


= ff, bts me banal)" de day = 0, (Rasy) # (a, m2). 
0 Jsn- 
The Fourier-Bessel coefficients of an integrable function are defined by 


Ap. = SB £2) Pk aml) 
ne Se. k,m(X)?dx 
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THEOREM 5a. Let f(x), x € Ba, be an integrable function for which the mean 


value 
Jens 1 f(rw)dw 


Signs dhe 
is piecewise C'™. 
i) Ifn=1,2, then limy <M Ak m@k,m(0) exists. 
wi) If n > 3 then limy—oo ei Ak mk,m(0) exists if and only if the mean 
(n—3) 
2 


ra jir)= 





value function f(r) has [ ] continuous derivatives for 0 <r < a which vanish 


atr =a. (|| = integer part.) 


REMARKS:. In case n = 1 the Fourier-Bessel series reduces to the ordinary 
Fourier series, which has been well-studied. In case n = 2 several authors have 
noted that, although the series converges at r = 0, the convergence may be 
expected to be slower than for r > 0. In fact, an explicit analysis of the Bessel 
function asymptotics reveals that for a piecewise smooth function with f(a) 4 0 
the speed of convergence at r = 0 is of the order O(M~—!/?) whereas at r > 0 the 
speed of convergence is O(M~'), just as for the one-dimensional Fourier series 
([GO], p. 34). When we pass to the three-dimensional Fourier Bessel expansion 
of such a function, we find [GP] that the series diverges boundedly at r = 0, just 
as for the previous three-dimensional examples. 


5b. General case: asymptotic solutions of differential equations. 
We now analyze the eigenfunction expansions on a general manifold with rota- 
tional symmetry. The Riemannian metric is written in geodesic polar coordinates 
as 
ds? = dr* + G(r)?d6" 
where G(r) is a smooth function with G(r) > 0 (r > 0),G(0) = 0,G’(0) = 
1,G”(0) =0. 
The equation satisfied by the radial eigenfunctions is 
G' 
0=Ad+Ad= ¢" (r) + (n~ 1a 


The asymptotic analysis for \ — oo has been carried out by Olver [(O, Theorem 
4.1, p. 444]. With some work, these can be expressed asymptotically in terms of 
®), which satisfies the Euclidean equation ®” + 2—16' + \® = 0 with 6(0) = 
With a suitable normalization for ¢,, we have 


Gv1 lon ar Ci 
oa(r) = (=)? “a(r) + Vr 


where lim)_,.. € = 0. In particular ¢,(0) = 1+ Ox =)s A — oo. For fixed r > 0 
) when A — oo. 


¢'(r) + Ad = 0. 


aes A — 00. 





the first term is O( rp) while the second term is O(—=¢ay Fe icaal =D 


This leads to the following theorem, exactly as for the Fourier-Bessel case. 

THEOREM 5B. Let f(r), 0 < r < a be a piecewise smooth radial function 
on a geodesic ball of an n-dimensional Riemannian manifold with rotational 
symmetry. Then the eigenfunction expansion 5°, Ardy, (0) converges iff r — 
f(r) has [233] continuous derivatives. 
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Appendix: proof of the classical result 


For completeness, we give a quick proof of the classical convergence theorem 
of Fourier transforms of smooth functions on R”. Without loss of generality, 
we prove convergence at x = 0, which can always be arranged by a suitable 
translation. 

We first prove convergence in case f(x) = e~!tl°/2. In this case a direct com- 
putation gives f(j) = e~!#!"/2/(2)"/?, so that fy(0) tee f(u) du =1. 

Next we consider f(x) = xjeltl’/2,1 <i <n. In this case a direct 
computation gives fi() = pje7!#!"/2/(27)"/2, so that fi,(0) =0 for all M and 
the required convergence also holds. 

Next, consider any integrable function for which f(x) = O(|z|?), |x| — 0. 
Define h(x) = f(x)/|z|*, also an integrable function. Its Fourier transform 
satisfies 
h(p) = (+)" fe.” h(x)dx, a C? function, whose Laplacian is 


1 
(20)" 





AR) == / le\e-#) h(a)de = — f(y). 


The spherical partial sum is transformed by the divergence theorem as follows: 


fu(0) = / oy 0 


=- / Ah(u)du 
|H| <M 


dh 
Ll= 


where we have used the smoothness of f in the last step. 
Finally, for the general case, write 


f(a) = fO)e FN? + 5a: f(O)ver "2 + F(z) 


w=1 


which defines F. Clearly F is also smooth and satisfies F(0) = 0, 0;F (0) = 0; 
hence F(x) = O(|z|*),|z| — 0. From the above argument we conclude that 
Fy4(0) — 0 when M — ov, hence fx(0) — f(0), which was to be proved. 


REFERENCES 


[A] M. Ash, Multiple trigonometric series, In: Studies in Harmonic Analysis, Studies in 
Mathematics, MAA, Washington DC. 

[B1} S. Bochner, Ein Konvergensatz fiir mehrvariablige Fourtersche Integrale,, Math. Z. 
34 (1931), 440 - 447. 

[B2] S. Bochner, Summation of multiple Fourier series by spherical means, Tran. Amer. 
Math. Soc. 40 (1936), 175 — 207. 

[CCTV] L. Colzani, A. Crespi, G. Travaglini and M. Vignati, Equiconvergence theorems for 
Fourier-Bessel expansions with applications to the harmonic analysis of radial func- 
tions in euclidean and non-euclidean spaces, Trans. Amer. Math. Soc. 338 (1993), 43 
— 55. 


152 MARK A. PINSKY 


GO] D. Gottlieb and S. Orzag, Numerical Analysis of Spectral Methods: Theory and Ap- 
plications, SIAM, Philadelphia, PA, 1977. 

GP] A. Gray and M. Pinsky, Gibbs’ phenomenon for Fourier-Bessel Series, Expositiones 
Mathematicae 11 (1993), 123 — 135. 

HL} G.H. Hardy and E. Landau, The lattice points of a circle,, Proc. Royal Soc., A, 105 
(1924), 244 — 258. 

O} F. W. Olver, Asymptotics and Special Functions, Academic Press, New York, 1974. 

P] M. Pinsky, Pointwise Fourier inversion and related eigenfunction expansions, Com- 
mun. Pure Appl. Math. 47 (1994), 653 - 681. 

PST] M. Pinsky, N. Stanton and P. Trapa, Fourier series of radial functions in several 
variables, J. Functional Analysis 116 (1993), 111-132. 

Sw] E. M. Stein and G. Weiss, Introduction to Fourier Analysis in Euclidean Spaces, 
Princeton University Press, Princeton, N.J., 1971. 





DEPARTMENT OF MATHEMATICS, NORTHWESTERN UNIVERSITY, EVANSTON, IL 60208 


E-mail: pinskyQ@math.nwu.edu 


Contemporary Mathematics 
Volume 191, 1995 


Total Positivity, Harmonic Analysis 
and Random Walks on Weyl Chambers 


DONALD ST. P. RICHARDS AND KENNETH I. GROSS 
Dedicated to Ray A. Kunze 


ABSTRACT. This paper continues our study of total positivity from a group- 
theoretic perspective. In earlier investigations we used techniques from 
harmonic analysis on compact Lie groups to establish total positivity of 
functions. In the present paper we indicate how the theory of random 
walks on Weyl chambers leads to combinatorial examples of totally positive 
functions and, more generally, how totally positive functions arise naturally 
in representation theory. 


1. Introduction 


This article is the third in a series of papers treating the concept of total 
positivity from the point of view of group theory and harmonic analysis. 

In the first paper [6], we related the classical notion of total positivity to 
analysis on the unitary groups U(n) and applied our techniques to obtain new 
positivity theorems for classical hypergeometric functions. In the second paper 
[7] we considered finite reflection groups and associated with each such group a 
new notion of total positivity. In contrast to the previous papers in which the 
results were primarily analytic in nature, the present paper has a combinatorial 
flavor. Here, we use the idea of a random walk on a Weyl chamber — a concept 
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that arises in statistics in the study of sequential data analysis — and a deep com- 
binatorial formula due to Gessel and Zeilberger [4], to construct totally positive 
functions. 

By way of background, we briefly review the classical notion of total positivity 
and its connection with the unitary groups. 

Let D C R? and r a positive integer. A function K : D —> R is said to be 
totally positive of order r (TP,) if the n x n determinants 


K(s1,t1) K(si,te) ++: K(s1,tn) 
Gy, - saeeaiy eter 
K(Sn,t1) K(sn,to) vee K(sn,tn) 
are nonnegative for all n = 1,... ,r and for alls; >--- > s, andt; >--- >t, 


such that (s;,t,) € D. If the determinants (1) are all strictly positive, then one 
says that K is strictly totally positive of order r (STP,). If K is TP, (STP,) for 
allr = 1,2,..., then we say that K is TP, (STP..). We mention in passing that 
there is a still stronger notion of total positivity called extended total positivity 
(ETP) which can be found, for example, in [6], [12]. 

Total positivity arises historically in the 1930’s in work of Schoenberg [16] 
and Gantmacher and Krein [2,3] on oscillation properties of integral operators, 
Sturm-Liouville theory, and the analysis of mechanical vibrations. It also has 
widespread applications in probability and statistics and in approximation the- 
ory. The principal reference is Karlin’s comprehensive treatise [12]. 

The central theme of our work in [6] is the intimate relationship between 
total positivity of multiplicative real-analytic kernels — that is, functions of the 
form K(z,y) = f(xy) where f : R — R is real-analytic — and analysis on the 
unitary groups. In particular, to any such kernel K we associate a sequence of 
real-valued functions w, of a matrix argument, having the property that for each 
dD 


(2) det(K(s;,t~)) = V(s)V(t) Wn (usu 't) du. 
U(n) 

Here, du denotes Haar measure on the unitary group U(n) normalized to have 
total volume one; s = diag(s,,...,8,) and t = diag(t,,...,t,) are diagonal 
matrices such that all products s;t, (j,k =1,...,m) lie in the domain of f; and 
V(s) = [h<jcren(s; — 8«) denotes the Vandermonde determinant. It follows 
from (2) that the function K is totally positive of order r if the functions ~, 
are nonnegative for n = 1,...,r. Conversely it can be shown that the total 
positivity of K implies that the functions w, are nonnegative. Thus the inte- 
gral representation (2) yields necessary and sufficient conditions for the total 
positivity of K (6). 

The context in which f is a classical hypergeometric function is particularly 
fruitful, as the functions w, are hypergeometric functions of matrix argument 
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[6,7] which, for appropriate values of the parameters and argument, can be shown 
to be positive. For example, when f is the exponential function, f(x) = e”, the 
associated functions are given by w,(£) = B,e§ for all n x n complex matrices 
£, where 3, = WaiG — 1)!, a positive constant. Here, (2) takes the form 


a det(es#**) 
3 / eft(usu t) du = Bn 
®) ie V(s)V) 


from which we see immediately that the function K(x, y) = e*¥ is STP... This 
is perhaps the most basic example of a totally positive function. 

As we will see in the next section, the association of the unitary groups with 
the classical notion of total positivity is far from accidental. Indeed, from an 
expanded Lie-theoretic perspective, the notion of total positivity is intrinsically 
group theoretic and formula (3) is the special case for the group U(n) of a general 
formula ((10), below) due to Harish-Chandra that is valid for all connected 
compact Lie groups. 


2. Total positivity for finite reflection groups 


Let W be a finite reflection (or Coxeter) group acting on R", by which one 
means a finite subgroup of the orthogonal group O(n) acting naturally on R” by 
orthogonal linear transformations. We denote the action of w € W on s € R” 
by w-s, and remark that det w = +1 for all w € W. The action of W partitions 
R” into a finite number of mutually congruent open subsets known as Weyl 
chambers. The boundary of a Wey] chamber is a union of hyperplanes called 
walls of the chamber. The group W then acts simply transitively on the set of 
Wey] chambers. We choose a fixed chamber C and refer to it as the fundamental 
Weyl chamber. For a complete treatment of finite reflection groups, and in 
particular a more explicit definition of the fundamental Wey] chamber in terms 
of a system of simple roots, see [9]. 


2.1. DEFINITIONS. Let D be a subset of R” x R” closed under the action of 
W x W, ie. (s,t) € D implies (wis, wot) € D for all wi, wo € W. 

(i) A function K : D — R is invariant with respect to W, or simply W- 
invariant, if K(w-s,w-t) = K(s,t) for all w € W and all (s,t) € D. 

(ii) For any W-invariant function K : D — R set 


(4) DwK(s,t)= S> (detw)K(w-s,t) 
wEew 
for all (s,t) € D, 
(iii) A W-invariant function K : D — R is totally positive with respect to W if 
(5) Dy K(s, t) 20 


for all (s, t) € DN(C xC), where C is the fundamental chamber. If strict inequality 
holds in (5) we will say that K is strictly totally positive with respect to W. 
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The alternating sum Dy is an analog for W of the usual determinant. In 
fact, when 


(6) =e (s;,t, 


for some function K of two real variables, then (4) takes the form 
n 
(7) Dw K(s,t) = 5 > (det w) es ), 
wew j=l 
which is more closely analogous to the ordinary determinant. Indeed, when W 
is the symmetric group, (7) is precisely det(K (s;,t;)). In [7] we restricted our 
attention to functions K of the form (6) for which the definition of Dw K in (7) 
was sufficient. However, for the functions treated in this paper the definition 
(4)is more appropriate. In the context of total positivity this definition was first 
written down by Karlin and Rinott [13], but the concept of an alternating sum 
is of longstanding importance in representation theory (cf. Section 4 below). 
The proposition that follows provides analogues for Dw of three basic prop- 
erties of the ordinary determinant: (i) If the rows or columns of a determinant 
are permuted then the sign of the determinant changes in accordance with the 
sign of the permutation; (ii) The determinant is identically zero if two of its rows 
or columns are identical; (iii) The Binet-Cauchy formula. The first two parts 
appear as Proposition 3.2 in [8]. Part (iii), in various forms, appears in [8], [12], 
[13] and [17]. 


2.2. PROPOSITION. (i) If w © W and (s,t) € D then 
DwK(s,w-t) = Dw K(w-s,t) = (detw)Dw K(s,t). 


(ii) If s or t belongs to a wall of a Weyl chamber then Dw K(s,t) = 0. 
(iit) Let L and M be W-invariant functions on Dp = D, x Do and Dy = 
D2 x D3, respectively. Let u be a W-invariant Borel measure on R". Define 


(8) K(s,)= ff L(s, z)M(z,t)du(z) 


on the set D of all (s,t) € R” xR” for which the integral is absolutely convergent. 
Then D is closed under the action of W x W, K is W-invariant, and 


(9) DwK(s,t) =f Dw L(s,z)Dw M(z, t)du(z) 


for all (s,t) € D. In particular, if L and M are (strictly) totally positive with 
respect toW then so is K. 


A large class of finite reflection groups, called Weyl groups, arises in the struc- 
ture theory of compact Lie groups. For such finite reflection groups, harmonic 
analysis on the compact group provides a tool for constructing totally positive 
functions. 
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Thus, let U be a compact connected Lie group, u its Lie algebra, g = ut+/—lu 
the complexification of u, and Bo(-,-) the trace form on g. Let T denote the 
maximal torus of U, t the Lie algebra of T, h = t+ ,/—It the complexification of 
t, and h* the dual space of h. Let A C h* be the root system for U, fix a basis 
of simple roots, and denote by A, the resulting subset of positive roots. The 
Weyl group W of U is the group generated by the set of simple reflections with 
respect to the roots. If we decompose t as t = 3 to where 3 is the center of u 
and tp is the semisimple part of t, then we can view W as a finite reflection group 
of orthogonal linear transformations on the real vector space tp (with respect to 
an inner product determined by the trace form). The dimension n of to is called 
the rank of U. We fix as the fundamental Weyl chamber C in to that chamber 
which is positive with respect to the system of simple roots. 

We can now state the following formula of Harish-Chandra [10] that general- 
izes formula (3) above. 


2.3. THEOREM. (Harish-Chandra) Suppose that Hi, Ha € ) and du denotes 
Haar measure on U with total volume 1. Then 
Bo(w-Hi,H 
(10) fee ee du = By ew et wye o( 1,H2) 
U V(A1)V (He) 
where Bw is a positive constant, and 


V(H) = [I o(#) 


acAy 
for any H Eb. 
2.4. COROLLARY. (([8], Theorem 6.1) Suppose that the finite reflection group 


W is the Weyl group for a compact Lie group U. Then the function K(s,t) = 
eS) st CR", is strictly totally positive with respect to W. 


To prove this result, simply identify elements H,, H2 € /—1tp with vectors 
s,t € R”. If s,t € C then, by Harish-Chandra’s theorem, 


DwK(s,t) = )> (detw)K(w-s,t) 
wEew 
S (det wes) 


(11) wEew 
ye (det w)eBo(w Hi .H2) 


wew 


= Bw 'V(H)V(Ha) f eBo(Ad(u) Ai, 2) gay 
U 


It is now evident that Dw K(s,t) > 0 for all s,t € C. Therefore, K is strictly 
totally positive with respect to W. 

If we define total positivity with respect to formula (7) (rather than (4)) then 
we can rephrase the corollary as the statement that the function K(z, y) = e?¥ 
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for (z,y) € R? is totally positive, not just in the classical sense, but also with 
respect to any Weyl group W. 


2.5. EXAMPLES. (1) When U = U(n) the Weyl group W is S,,, the symmetric 
group on n symbols, and total positivity is the classical notion. 

(2) For the orthogonal group U = SO(2n +1) the Weyl group W is generally 
denoted by G(n) and can be described as follows: Each w € W is of the form 
w = eo, where o € S, is a permutation on n symbols, € = (€1,...,€,) with 
€é; = +1 for all j = 1,...,n, and w = eo acts on s = (81,...,5n) € § by 
ws = (€185(1);--- »€nSo(n))- In particular, det(w) = €1 --- en (sgn). 

For U = SO(2n + 1) we identify t) with R”, and C with the collection of 
elements s = (s1,.-.,5n) € R” such that s) > so--- > s, > 0. Lets = 
($1,--.,Sn),t = (t1,...,tn) € R”. Upon simplifying the sum over W on the 
right-hand side of (11), we deduce ((8], Section 6) that the strict total positivity 
of the function K(s,t) = e(*:*) is equivalent to positivity of the n x n determinant 
det(sinh(s;t,)) for all s; >--- >s, >Oandt,; >--- >t, > 0. Therefore the 
strict total positivity of the function K is equivalent to the well-known result 
[12] that the function K(x, y) = sinh(xy), x,y > 0, is strictly totally positive in 
the classical sense. 


2.6 CONJECTURE. We conjecture that the function K(s,t) = e'**), s,t € R®, 
is strictly totally positive with respect to all finite reflection groups. 


In view of Corollary 2.4, only the case in which W is not a Weyl group remains 
to be discussed. The following example indicates the difficulties to be addressed 
in resolving this conjecture. 

Let W = H?, the dihedral group with n reflections, acting on R?. Although 
W is a finite reflection group, only for n = 1,2,3,4 and 6 is W the Weyl] group 
of a compact Lie group. Working with polar coordinates on R?, we choose as 
the fundamental Weyl chamber the region 


nt ; (n — 2)r T 
c= {(n0)ir>0, ya <O< 50. 





Then the criterion that the function K be strictly totally positive with respect 
to W reduces ([8], Section 6) to the inequality 


(12) S > Glin(r) sin j0 > 0 
j=l 


for0 <6<7,r> 0. Here, I;(r) denotes the Bessel function of imaginary 
argument and of order j. 

The inequality (12) remains unproved for general n. However, the inequality 
is valid for n = 1,2,3,4,6 since, for those values of n, the dihedral group is the 
Weyl] group of a compact Lie group, so that Corollary 2.4 applies. Moreover, 
the cases n = 1,2 have also been established using classical results for the Bessel 
functions. 
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3. Random walks on Weyl chambers 


The study of random walks and lattice paths is an old and venerable area of 
probability, statistics and combinatorics. In statistics, random walk problems 
arise in sequential binomial sampling. A lucid description of these problems, and 
their application to estimating the size of certain animal populations, is provided 
by Lehmann ((15], p. 92). 

In the classical ballot problem, it is required to determine the probability 
that at no point in time did the winning candidate in a two-candidate electoral 
contest trail the loser [1]. This problem is solved by counting the number of 
ways of walking from the origin to a lattice point (mi,me2) with m,; > mz, 
taking positive unit-length steps in such a way that the walker always remains 
in the region {s; > sg}. Here, each vote for the winner is identified with a 
horizontal step while each vote for the loser is identified with a vertical step. 

More general ballot problems require enumeration of the number of ways of 
walking from the origin (0,... ,0) to a point (m1,... ,mn), with steps of length 
1 in the positive coordinate directions, and such that all steps lie in the region 
$8, > ++: > 8, > 0. This problem has been solved by several authors in various 
ways. A particularly elegant solution has been given by Zeilberger [18]. The 
result is as follows. 


3.1. THEOREM. (Frobenius; MacMahon; Zeilberger) The number of lattice 
paths from the origin to the point m = (m,... , Mn) taking positive unit-length 
steps, and with travel restricted to the region s; > --- > S, > 0, is 
Tie; (m = Mit +9) 

The appearance of these numbers, which are familiar in representation theory 
as Weyl’s degree formula, may seem surprising in this context. One explanation 
for their appearance begins with an analysis of the special case n = 2. In that 


case it may be verified using a simple graphical argument that F'(m , m2) satisfies 
the recurrence relation 


F(m1,m2) ={ 


(13) F(m) = (mi +++» +mn)! 


F(m, —1,m2)+ F(m1,m2—-1), mi > me 
F(m,,mz2 —- 1), my, = m2 
Starting with the initial condition F(0,0) = 1 the recurrence relation can be 


solved readily to yield the result in Theorem 3.1. 
This technique extends to the case of general n. Indeed, if e, = (1,0,... ,0), 


eg = (0,1,0,... ,0), ..., en = (0,... ,0,1) denotes the standard basis for R”, 
then the number of paths F'(m) satisfies the recurrence relations 
(14) F(m) = ) 7 F(m—e;) 

J 


where the sum is taken over all 7 such that m — e; remains in the region s; > 
“++ > 8, > 0. As before, we also have the boundary condition F(0,... ,0) = 1. 
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As it turns out, these recurrence relations are implicit in the work of Johnson 
[11], arising from the methods applied in [11] to decompose tensor products of 
the elementary representations of GL(n,C) (cf. Zhelobenko [19]). Specifically, 
it may be deduced from [11] that the recurrence relations (14) characterize the 
degrees of the irreducible polynomial representations of GL(n, C). Therefore the 
numbers F(m) are given explicitly by (13). 

It appears that these remarks constitute a new solution of the ballot problem, 
admittedly less elegant than the solution given by Zeilberger [18]. 

We turn now to the problem of constructing totally positive functions through 
the technique of ballot problems. Here, we shall apply a deep combinatorial 
formula of Gessel and Zeilberger [4] and will work within the context of abstract 
root systems. 

Thus, following [4] and [5], let A C R” be a finite root system, W be its 
associated finite reflection group, UY be a base for A, and denote by C the funda- 
mental Weyl chamber. Let L be a lattice embedded in R”, and assume that L is 
invariant under W; i.e., w- L = L for all w € W. Denote by Sa set of allowable 
steps, by which we mean a finite subset S of L that is also invariant under W. 

We also assume that walks are reflectable, meaning that the following two 
conditions hold: (1) Any step s € S from a lattice point in the interior of C 
cannot exit C; and (2) for each simple root a € W, there is a real number ky 
such that (a, s) = +k, or 0 for all s € S, and (a,/) is an integer multiple of k, 
for all] € L. 

The condition of reflectability ensures that a random walk cannot escape the 
fundamental chamber C without first hitting a wall of C [5). 

For any positive integer r, and any s,t € L, define K: L x LR by 


K,(s,t) = WALK,(s,t), 


the number of paths from s to t using exactly r steps within S, and unconstrained 
by a chamber. Then by a fundamental result of Gessel and Zeilberger [4], we 
deduce the following result. 


3.2. PROPOSITION.. The function K, is totally positwe with respect to W; 
that is, Dw K(s,t) > 0 for all s,t € L. 

As with the results in Section 2, Proposition 3.2 is established by finding an 
explicit formula that makes clear the nonnegativity of Dw K on L x L. In [4], it 
is proved that Dw K(s, t) is exactly the number of r-step paths from s to t that 
always stay within the interior of the fundamental Weyl chamber C. This yields 
nonnegativity immediately, proving that K is totally positive with respect to W. 

In closing this section, we note that the results in [4] are also established for 
affine root systems. 


4. Concluding remarks 


As noted at the outset, this article adopts the point of view that harmonic 
analysis and representation theory provide natural settings in which to frame 


th 
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e concept of total positivity. Indeed, nonnegative alternating sums that arise 


naturally in representation theory may be interpreted as establishing the total 
positivity of particular functions. For example, Weyl’s character formula and 
Kostant’s partition function [14] are both alternating sums in the sense of (4). 


In 
th 


10. 


11. 


12. 
13. 


14. 


15. 
16. 


17. 


18. 


19. 


both cases the total positivity of the underlying function is immediate from 
e representation theory. 
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ABSTRACT. Let K be maximal compact in SU(p,q) and let g = Ap+q-1- 
Covariant differential operators are used to identify nontrivial unitary ir- 
reducible (g, K) modules in the family F = {I*(N(A)*) : i > O} where 
I denotes the Zuckerman functor and N(A)* denotes the conjugate dual 
of a generalized Verma module N(A) whose irreducible quotient is unitary 
and not N(A). The nontrivial module [4(N(A)*) in F is identified here 
as unitary and irreducible, has cohomological degree I below the middle 
dimension, and is of highest weight type. The covariant differential op- 
erators used to make this identification arise as differentials in a spectral 
sequence induced by a Bernstein-Gelfand-Gelfand resolution of N{A). A 
Bott-Borel-Weil result is obtained by producing a Weyl group element o of 
length I such that the irreducible unitary module [4(N(A)*) has highest 
weight o(A + p) — p. 


1. Introduction 


Derived functor induction is a general method for constructing admissible rep- 
resentations of a real reductive group G. The method uses the derived functors 
I’, i > 0, of the Zuckerman functor I’ to produce (g, K) modules where K is 
a maximal compact subgroup of G and g denotes the complexification of the 
Lie algebra go of G. One begins with a (g, 4) module V where M is a sub- 
group of K and considers the family F(V) = {I*(V) : i > 0} of modules. The 
problem is to identify which of the nonzero modules in F(V) are unitarizable 
and irreducible. This problem becomes more tractable if one begins with (g, M) 
modules V that are well understood and whose known structure one can readily 
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exploit. For this reason it is reasonable to take V to be a generalized Verma 
module or its irreducible quotient. This is the strategy taken in [9], [14] and [20] 
and is the strategy we employ here as well. However, our method for identifying 
irreducible unitary modules in F(V) differs from the splitting criteria methods 
found in [9] and [14]. In this article we utilize covariant differential operators 
to make the desired identifications. The differential operator approach taken 
here allows us to prove a Bott-Borel-Weil result in which the derived functors I° 
yield irreducible unitary highest weight (g, K) modules where K is a maximal 
compact subgroup of SU(p,q) and g = Apig-1. We develop some notation and 
more fully describe our methods and results. 


Let p,q > 1 be integers and set n = p+q. Let G = SU(p,q) and let G’ 
denote either SU(p+1,q—1) or SU(p—1,q4+1). Let K and L denote standard 
maximal compact subgroups of G and G’, respectively. Set M = LOK. Let 
Qo denote the Lie algebra of G and let g denote the complexification of go. We 
make the standard choices for Cartan subalgebra t and set {a;:1<i<n-—1} 
of simple roots. Let w; denote the fundamental dominant weight corresponding 
to the simple root a;. If G’ = SU(p+1,q—1), let q be the parabolic determined 
by wp+1 and if G’ = SU(p—1,q+1), let q be the parabolic determined by wp_1. 
Let S denote the middle dimension determined by q and K. Let Fy,(A) be an 
irreducible L module with highest weight \ and set N(A) = U(g)@uq) Fx (A). We 
let N(A)* denote the conjugate dual of M(A) and let L(A) denote the irreducible 
quotient of N(A). 


Let A(G’) be the set of highest weights for unitary highest weight representa- 
tions of G’. The set A(G’) was completely determined in [8]. Define the subset 
A,(G’) of reduction points by A,(G’) = {A € A(G’) : N(A) is reducible}. For 
d € A,(G’), consider the family F(A) = {T*(N(A)*) : i > 0} of (g, K) modules. 
We study the family F(A) using a spectral sequence induced by a Bernstein- 
Gelfand-Gelfand (BGG) resolution of N(A)* (cf. section four). The E> terms of 
the spectral sequence are computed in section five. From these terms one sees 
that for a certain index J the module I'/(N(A)*) can be completely determined 
by the resulting FE, terms provided I < S. Specifically, there exist integers J 
and N such that J+ N = J and EZN =I! (N(A)*) if J < S. The integers J and 
N depend only on the cone in which A lies. We refer to section six of [5] for the 
description of the cone decomposition of A,.(G’). We next show that E2% arises 
as the kernel of a differential d/’% of the spectral sequence. This differential d2- 
is related to a (g, A’) invariant differential operator called a covariant differential 
operator (CDQ). Since CDOs are central to our method we now give a brief 
description of these operators and indicate how they are used. 


Let (G,K) denote a Hermitian symmetric pair and let U, and U,, denote 
holomorphically induced representations of G. A CDO D,,, : U, — U, is 
a continuous intertwining map. We refer to [6] for the explicit form of D,,,, 
in standard realizations of holomorphically induced representations. A certain 
distinguished class of nonzero CDOs has the special property that the space of K 
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finite vectors in Ker(D,.,,,) is an irreducible unitary highest weight (g, K) module 
(cf. (3.2) [6]). 

With this distinguished class of CDOs in mind, we outline our strategy. Since 
the space of such intertwining maps is one-dimensional (cf. [2], [3]) the operator 
D-,,, is completely determined by the pair (7, ). In addition, all pairs of weights 
(r,u) for which Ker(D,,,,) is irreducible and unitary are explicitly listed in [5]. 
We can therefore recognize a differential in a spectral sequence as (a multiple 
of) a distinguished CDO D,,,, by identifying the appropriate parameters (7, i). 
Moreover, if it can be shown that a nonzero CDO appears, then the irreducibility 
and unitarizability of the kernel follow immediately. 

In section six we identify the distinguished CDO pair (7, 1) for which d?-"" = 
cD,,,¢ € C, and give the associated cone data. In particular, the dependency 
of the pair (7, 4) on the form of the weight is made explicit. We view this 
dependence as relevant because it measures, to some extent, the manner in which 
CDOs factor through the induction process. After all, the assumption A € 
A,(G’) implies that there exists a distinguished CDO D) ,(,) whose kernel in 
N(A)* is an irreducible unitary (g, L) module. The transfer of unitary structure 
by cohomological induction is reflected in the CDOs that appear as input and 
output. 

The strategy outlined above is efficacious provided one can show that the 
above differential is nonzero. To this end, define the subset A¥(G’) of A,(G’) 
by the condition that the irreducible L module Fy,(A) remains irreducible upon 
restriction to the subgroup M. We show in section seven that for \ € AM(G’) 
the differential d7" is nonzero. By the special property of these distinguished 
CDOs we conclude I(N(A)*) is an irreducible unitary highest weight (g, K) 
module with highest weight 7. The Bott-Borel-Weil result mentioned above 
is established in section five where it is shown that there exists a Weyl group 
element o with (0) =I and r =o(A+ p)- p. 

Explicit computations for the case G’ = SU(p+1,q-—1) are presented in the 
body of the paper and analogous results for G’ = SU(p — 1,q + 1) are left to 
the final section. We now turn to some results in the literature related to this 
article. 

In [14] more general Hermitian symmetric pairs (G’, L) and (G, K) are studied 
and it is shown that for certain 4 € A,(G’) the family {I*(L(A)) : i > 0} 
contains irreducible unitary (g, K) modules that are not highest weight modules. 
However, in order to satisfy a certain splitting criterion, \ is taken as the highest 
weight of a one-dimensional module F,(A). Related results are obtained in [20] in 
a more general setting where (G, K) is no longer Hermitian symmetric but where 
the Cartan involutions @ and 6’ corresponding to real forms G and G’ commute. 
Certain unitary derived functor modules are explicitly computed in terms of their 
K types. These modules are examples of nonhighest weight modules and the 
condition dim(F,(A)) = 1 is imposed to identify the K structure. In contrast, 
we produce unitary highest weight modules and there is no constraint on the 
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dimension of Fy, (A). 

Spectral sequences are a common tool in the computation of sheaf cohomol- 
ogy. For example, spectral sequences where used in [7] to show that solutions 
spaces of linear massless free-field equations can be realized as cohomology on 
twistor space. Generalizing this setting, the Penrose correspondence views sheaf 
cohomology on one flag manifold in terms of kernels and cokernels of invariant 
differential operators on sections of homogeneous vector bundles on a second 
flag manifold. We refer the reader to [1] for explicit details as well as many 
illustrative examples. The role of the Bernstein-Gelfand-Gelfand resolution and 
the resulting spectral sequence stands out in [1] as a dominant feature of this 
correspondence. 

With these observations in mind, one can view aspects of what we do here 
as an algebraic version of the Penrose correspondence. Sheaf cohomology in 
the Penrose correspondence is replaced here by derived functor modules. The 
connection between analytic cohomology and the Zuckerman functors is well- 
known and is discussed in chapter eleven of [1]. What is new here is the role 
that covariant differential operators play in determining the irreducibility and 
unitarity of the cohomologically induced modules. 


2. Preliminaries 


Ip 0 : 
been Setting n = p+4q, 


we put SU(p,q) = {g € GL(n,C) : g*Ip.qg = Ipq} where * denotes conjugate 
transpose. We choose as maximal compact subgroup K of SU(p, q) the subgroup 


Let p and q be positive integers and let I,,4 = 


Ke a z| A € U(p), B € U(q), det(A) - det(B) = i}. 


Let 0(X) = —X* where X € C"*". The Lie algebra go of SU (p,q) decom- 
poses according to the eigenvalues of 8. Specifically, the subalgebras 


fo = lo i :a€ CP*P be C1*4, a* = —a, b* = —b, tr(a) = -1n()}, 


= Oe]. Pxq 
we hee 


correspond to the 1 and -1 eigenspaces, respectively. We drop the subscript “0” 
to denote complexification of a Lie algebra. Complexifying 6, we obtain the 
Cartan decomposition g = § @ p of g = An_; where 
a 0 . axg 
t= 0 6 :a € CP*? be C%*4 tr(a) = —tr(b) 


and 


p= 4[i 4 cee de crt. 
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00 
denotes conjugation in g with respect to go. 


_ We also define subalgebras p, = {[° ;| “EE cra} and p_ = p, where bar 


Let to denote the diagonal traceless matrices in go. Then t is the subalgebra 
of diagonal traceless matrices in g and the corresponding set of roots of g with 
respect to t is A = {e; — e; : i # j} where e; € t* selects the 7° entry of the 
diagonal. We take {a; = e; — e;4; : 1 <i <n-—1)} as the set of simple roots. 
The corresponding set of positive roots is At = {e; —e; : i < j} and we let 
{w;:1<i<n-—1} denote the associated set of fundamental dominant weights. 

A linear functional 7 € t* that takes purely imaginary values on tg is called 
a weight. We follow (5.2.1) [18] and define parabolic subalgebras q = q(w) of g 
for each weight w. Let g. denote the root space corresponding to a € A. Define 
subsets of A by 


A(é) = {a € A: (a,w) = 0}, 
A(u) = {a € A: (a,w) > 0}, 
A(q) = {a € A: (a,w) > 0}, 


and corresponding subalgebras by 


e=to( @ Ba); u= @ ga, g=l@u. 
ac A(e) acA(u) 


Then q is a parabolic subalgebra satisfying 0(¢) = @, 0(u) = u, 2 = @ and 
q = (qNt) @ (qMp). The decomposition q = ¢ @u is called the 6 stable Levi 
decomposition. Note that the parabolic q(w;) is defined for each fundamental 
dominant weight w;, 1 <7<mn-—1. In particular, we observe that § @ p, is the 
6 stable Levi decomposition of q(w,). 

In this article we will be concerned with the two parabolics q(w,-1) and 
q(wp+1). In order that these be defined, we henceforth assume that either p > 2 
or g > 2. In the sections that follow, we shall give explicit calculations using 
only the parabolic q(w,+41) and reserve the notation @ and u for this particular 
parabolic. The results for the parabolic q(w,_1) are similar and we state these 
results in section eight, leaving the omitted details to the reader. 

For later reference we explicitly determine the parabolic subalgebra q 
G(wWpii). We clearly have A(é) = {t(e;-—e;): pt+tl<i<j<norl 
i<j < pt} and A(u) = {e,-—e,;:1<icpt+landp+2 <j <n} wit 
corresponding algebras: 


Ss lA ll 


0 5b 


0 «2 
= P (p+1) x (q—-1) 
w= 4/5 “| :2EC \. 


Let Z(é) denote the center of @ and set @5° = [@, 4]. Then ¢ = @°* 6 Z(é) as the - 


= {6 | : tr(a) = —tr(b),a € CPtY* P+) be cece} 
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direct sum of ideals. Specifically, we have 


a 0 
ss : ne -_ “(p+1)x(p+1) (g-1)x(q-1) 
e ={(9 5 | tla) =e) =0,a€C EC y 


~ 


so that Z(¢) C t, dim(Z(é)) = 1 and é** = A, @ Ag_2 is semisimple. 

We now consider the parabolic q; = q(wp + w p41) and denote the Levi 
decomposition as qi = m@uwu,. It is clear that A(m) = A(é) M A(t) and 
A(u,) = A(u) UA(p,) so that m = @Met, uy) =ut+py and qi = q(wp+1) MN q(wy). 
Set m°* = [m,m] and let Z(m) denote the center of m. Since m has the block 
form 


0 0 
b 0] :aE CP? ce CU-DX-)), _b = tr(a) + tr(c) } , 
0 c : 


3 
I 
ooes 


we find that the decomposition m = m** © Z(m) takes the form 


a 0 0 
m>=<¢10 0 0] :ae€CP*?,ce C-)*4-) tr(a) = tr(c) = 0}, 
0 0 ¢ 
al, 0 0 
Z(m) = 0 —(pa+ (gq —1)d) 0 :a,bEC 
0 0 bIg-1 


~ 


In particular, we see that Z(£) C Z(m) C t, dim(Z(m)) = 2 and m** = A,_1 © 
Ag-2 is semisimple. Note that the simple system of roots S(m) for the set of 
positive roots A*(m) = A+ mM A(m) of mis S(m) = {a; :7 4 p,p+ 1}. 

Let L denote the maximal compact subgroup of SU(p+ 1,q — 1) given by 


(2.1) z= {(¢ BAe P+ 1), Be Ula M)pdet(A) -der() = 1}. 


It is clear that the complexification of the Lie algebra of Lis ¢. Weset M = LOK. 
Then 
(2.2) 


“(f 


In particular, note that M = U(p) x U(q) is connected and that the complexified 
Lie algebra of M is m. 


0 
a : AE U(p),B Ee U(q—-1),c € C,c- det(A) - det(B) = i 
B 


on oO 


3. Spectral Sequences 


We briefly describe some standard results in homological algebra which pro- 
vide the basic tools to compute cohomology. Details may be found in chapter 
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fifteen of [4] or chapter eleven of [16]. The goal of this section is to use an appro- 
priate resolution of a complex to construct a spectral sequence which converges 
to the desired derived functor module. We begin with a definition. 

A proper injective resolution M of a complex N :0 — N° > N! >... is 
a double complex M = {M™* : r,s > 0} of injective modules with commuting 
diagram 








0o- Mol = Mil a M21 “ 
* 1 : 
(3.1) 0 > Mo -, Mio _, M20 _, 
% 1 : 
0—>— N —~ MN -—+ N = 
“ : ie 
0 0 0 


such that, for each m > 0, the m** column M™* of M is a deleted injective 
resolution of N™ and each row in (3.1) is a complex. Furthermore, if M*’”" 
denotes the m*® row and if 
ZUM — Ker - Ym™ an Mrtim 
a > 
Be™ = im: Me TL” yn 
7 ? 
He™ = gam pam 


denote the cocycles, coboundaries and cohomology of the m** 


following complexes are injective resolutions: 
0 — ZN) > Zr? - Zr i, 
0 — BN) — Br | Bri 
0 — H"(N) — H”® =| Am 


row, then the 


By the dual of Lemma 11.33 [16], if C is a category of modules having enough 
injectives and if N is a complex, then N has a proper injective resolution. 


(3.2) PROPOSITION. Let TI denote a covariant left exact additive functor de- 
fined on a category C of modules having enough injectives. Let A be a module in 
C and suppose 


(3.3) 03 ASN GN! BO... 


is a resolution of A in the category C. Then there exists a first quadrant spectral 
sequence with 
Bo SN he A) 
nr 


where N denotes the deleted compler 0 — N° > N! >..--. The differential 
d,: E, > E, has bidegree (r,1—r). 
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Proor. Let M be a proper injective resolution of N. The exact sequence 


mL, 7m 
0—- Zum _, Mme, Brtim +0 





splits since Z™” is injective. Likewise, the exact sequence 0 > B™™ 3 Z™™ _, 
H™™ _, 0 splits since B™™ is injective. Since the functor [ is additive, it 
preserves split exact sequences. From the split exact sequence 


oss Ze) =n (mM) ra") r(Brthm) =si0), 
we have Ker(['(d™™)) =T(Z™™) and im(I'(d™™)) = T(B"+!)™). Consequently, 
Ker(C(d™™))/im(I?(d"-}™)) = 1(Z™™)/T(B™™). From the split exact se- 
quence 


~ 


we have ['(2™™)/T(B™™) =T(H4™™). But then one has 


~ 


(3.4) H™(0(M*"")) = Ker(P(d"™))/im(P(d"-»'™)) = P(H™™), 


We now consider the double complex [(M) = {['(M™) : r,s > 0} and 
cohomology of the associated total complex T(['(M)). We begin by determining 
the /, terms of the spectral sequences arising from the first and second filtrations 
of T(T(M)). 

The E terms ‘E;’” arising from the first filtration are obtained by initially 
taking cohomology up the columns of [(M), obtaining a double complex whose 
mh row has the form 


OS FON PRN POA) Ss 
since the columns of M form injective resolutions. One then takes cohomology 
down the rows to obtain ‘Ey’" = H"(['™(N)). 
The E, terms “E;””” of the spectral sequence arising from the second filtration 


are obtained by initially taking cohomology down the rows of I'(M) thereby 
obtaining a double complex whose n** column is of the form 
0 H"(I(M™)) = H"(I(M*™")) > --- 
and then taking cohomology up the columns. However, by (3.4), the n*® column 
is 
PGE) 50 CHS ie RU) ni 
Since the modules H"” form an injective resolution of H"(N), we have 
A™(T(H™")) =T™(H"(N)) by definition of the derived functor [” of [. But 
(3.3) is a resolution of A and so is exact. Therefore, we have 
POP IN) SEA) jr0 
rein) = { : 
0 ifn > 0. 
Since “EQ”” = ['™(H"(N)) = 0 for n > 0, the second spectral sequence collapses 


and we have 
H™(T(C(M))) = “By? =" (A). 
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It now follows that 
‘Ey = A"(0™(N)) => A™*™(T(E(M))) =0"F™(A). 


Moreover, the differential d, has bidegree (r,1 — r). This completes the proof. 


4. A BGG resolution 


The goal of this section is to choose a convenient (g,4/) module A and con- 
struct a suitable resolution of A by (g, 4) modules in order to apply (3.2). We 
refer the reader to section 6.1 of [18] for the precise definition of the category 
C(g, M) of (g, M) modules and for basic properties that this category enjoys. In 
sections four through seven we assume g > 1 and work entirely with the parabolic 
q = q(wp+1), leaving the results for the parabolic q(w,_1) to section eight. 

The construction of our desired resolution requires several steps which we 
now outline. We begin with a representation space F;,(A) for an irreducible 
representation of the compact group L in (2.1) with highest weight \. We first 
apply the construction in [13] to the semisimple Lie algebra °° in order to obtain 
a resolution of F,(A\) by £°5 modules. We then show that this resolution extends 
to a resolution of F',(A) by (€,M) modules. Finally, by applying appropriate 
exact functors, we obtain a resolution of the conjugate dual of a generalized 
Verma module by (g, /) modules. 

Following [13], we begin with a choice of a subset S of simple roots of the 
simple roots of @°°. This choice determines the parabolic subalgebra Ps of €°° 
from which we construct generalized Verma modules on which ¢** acts. We take 
S to be the subset S(m) = {a; : i # p,p +1} of simple roots of m. Then the 
simple Lie algebra generated by g+%, a € S(m), is m®* given in section two. We 
set 

Us = ® Ga, ts =m? + (tM l*) and Ps =ts us. 
ac At (£)\A(m) 
Then from [13] one knows that tg is a reductive subalgebra of °° and Pg is 
parabolic subalgebra of @°°. We may see this directly by noting that tm @°° = 
spanc{h; : 1 4 p+ 1} where h; denotes the diagonal matrix with 1 in the it 
entry and —1 in the (+1)** entry. Consequently ts = m°* @C h, is of the block 
diagonal form 


ts = 


-OoOos 


0 0 
b O| :aECP*?,ceCU-YxG-) tr(a) = —6, tr(c) = 0 
0 c¢ 


The subalgebra ts has a one-dimensional center Z(ts) spanned by 
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One has Z(tg) C Z(m) and ts = m**@Czp. In particular, rg is clearly reductive. 
Set A(us) = A*(€)\ A(m). Then clearly A(ug) = {a; +---+tap:1< 7 <p} 
and us has the explicit form 


0 -- O 2y41 0 ++ 0 
z. 
(4.1) ug=|- Pas, | sajpar €C1< 5 <p 
; 0 : 
Qe 08 Oh. 20 ee 6 


From (4.1) one has ug = €M py. 

Recall that a weight 7 is said to be At(m) dominant (resp. m integral) if 
2(r, a)/(a, a) is nonnegative (resp. an integer) for all a € A*(m). For such 
a A*(m) dominant, m integral weight 7 € t*, there exists an irreducible m°° 
module F's(7) with highest weight 7. Define a Ps action on Fs5(r) by letting 
us act by O and letting the central element zo of tg act by 7(zo). Define an °° 
module Ns(r) by 


(4.2) Ng(7) = U(E*) @ups) Fs(r). 

Let {o; = 0a,,1 <i <n-—1} denote the set of simple reflections and let W 
denote the Weyl group of g. Let W (resp. Ws) denote the subgroup generated 
by {o,:iApt+1} (resp. {o;:i A p,p+1}). We set 
(4.3) Ws = {w © We: At(m) C wAt(2)}. 


For w € W, we set At(w) = {a € A*+ : w!a ¢ A*} and define the length of 
w, &(w), to be the number of elements in At(w). Let pg denote half the sum of 
the roots in A*(é). 

Let the weight A be A*(¢) dominant integral, ie. 2(A,a)/(a,a) is a nonneg- 
ative integer for all a € A*(é). It is well-known that w(A) is A*(m) dominant 
integral for all w € Ws. Indeed, sending A*(£) dominant weights to At(m) 
dominant weights is a property that completely characterizes the set W§ (cf. 
(3.2.2) [18]). It follows that the module Ns(w(A + pe) — pe) is defined for all 
w € Wé. 

We now state Theorem 4.3 [13] as it applies to the semisimple Lie algebra ¢°° 
and parabolic Ps. Note that from (4.1) we have dim(us) = p. 


(4.4) THEOREM [13]. Let A be an A*(€) dominant integral weight and let 
Fyss(A) denote the irreducible °° module with highest weight . Set 


Cj = Il Ns(w(A+ pe)— pe), for0<j <p. 
wEew!] e(w)=j 


Then there exist °° module maps d;,1 <j <p, and € such that the sequence 


(4.5) 0303 SSS 
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is exact. Here € is the canonical €°° module epimorphism. 


We refer to the resolution in (4.5) as the Bernstein-Gelfand-Gelfand (BGG) 
resolution of Fyss(A). In our setting the exact sequence (4.5) simplifies. One 
readily computes W4é to be 


— 
Wey ={1) Gp, OeFpats- > Sp? O14} 


This is most conveniently done by using the algorithm on page 41 of [1]. In 
particular note that Card(W$) = p+ 1 and the elements of W4 are determined 
by their length. We then have 


Co = Ns(A), 
C; = Ns(op-+- Op—j4i(At pe) — pe)), for l <j <p. 


It is the simple form of each C; that motivated the choice of parabolics q(w,+1) 
and q(wp-1) in section two. 

We now extend the sequence (4.5) to a sequence of (€, 4) modules. To do 
this we need to begin with analytically integral weights + and corresponding 
representations of M. A linear functional 7 € t* is said to be analytically integral 
if r(H) € 27iZ whenever H € ty satisfies exp(H) = 1. Clearly an analytically 
integral functional is a weight. Since M = U(p) x U(q— 1), M is a compact 
linear connected reductive group. By (4.28) [11], if7 € t* is analytically integral 
and A*(m) dominant, then there exists an irreducible representation 7, of M on 
Fy (7) with highest weight 7. Now each z € Z(m) acts by T(z) on Fyy(r). Since 
the central element zp of tg lies in Z(m) it follows that Fy,(7) is an irreducible 
mS* module on which zp acts by 7(zp). Consequently, the @°* module Ng(r) in 
(4.2) is defined. We put an (¢, M) action on Ngs(r) by letting M act by Ad@7, 
and letting Z(¢) act by r(z), z € Z(@). 

We may alternately view the (¢,M) modules Ns(r) as follows. Define sub- 
algebras te = ts @ Z(€) and Py = te Bus. Since Z(¢) 6 Czp = Z(m), we have 
te =m. Since m = €M€ and us = Mp, one has Py = £M q(w,) so that Py is 
a parabolic subalgebra of @. The space F'y4(T) is a Py module if we again define 
us to act trivially. Define an M action on U(¢) @u(p,) Fu (rT) by Ad @7,. It is 
then clear that as (¢, 1) modules we have 


(4.7) Ns(r) = U(€) @u(p,) Fa(r). 


(4.8) PROPOSITION. Let » be an analytically integral weight. Then for each 
a © We, the weight o(A + pe) — pe is analytically integral. Furthermore, if 
z€ Z(é), then one has (o(A+ pe) — pe)(z) = A(z). 


ProoF. The first statement follows by induction on é(¢). One uses that A+/¢ 
is 2 integral and that each a € A(é) is analytically integral. We omit the details. 
Since dim(Z(¢)) = 1, it suffices to verify the second statement for a nonzero 
z € Ze) Mito. Normalize z so that 7(z) = (7,w +1) for all weights 7. Since each 
element of W, fixes wp+41, we have (a(A + pe) — pe)(z) = (o(A+ pe) — pe, Wp41) = 
(A+ pe, o7! (wp41)) — (Pe, p41) = (At pe, Wp41) — (00, p41) = (A, wpt1) = A(z). 


(4.6) 
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(4.9) CoroLuary. If A € t* is analytically integral and A*(£) dominant, 
then for each j € {1,... ,p} the weight A; = op --- Op—541(A+ pe) — pe is ana- 
lytically integral and At(m) dominant. 


Proor. The weight A; is analytically integral by (4.8) and is A*(m) domi- 
nant since dp---0p—j+1 lies in Wé. 


(4.10) PROPOSITION. Let 7, and T2 be analytically integral and A*(m) dom- 
inant. Assume d : U(£) ®u(p,) Fu(n1) — U(£) @ucp,) Fu (12) is an £ module 
map. Then d is an M module map as weil. 


Proor. Let 7m (resp. 72) denote the representation of M on Fyy(7,) (resp. 
Fy(T2)). We need to show that 


(4.11) d(Ad(g) ® 71(9)(41 ---£, @ v)) = Ad(g) ® T2(g)d(£,--- 2, @v) 


for €,,...,€ € @v € Fu(m) and g € M. Since d is an £ map, the left side 
of (4.11) is (Ad(g)é1)---(Ad(g)é,)d(1 ® m1(g)v) and the right side of (4.11) is 
(Ad(g)é1)--- (Ad(g)é,-)(Ad(g) ® 72(g)(d(1 @ v))). It therefore suffices to show 
that 


(4.12) d(1 ® m1(g)v) = Ad(g) ® m2(g)(d(1 ® v)) for all g € M and v € Fy(71). 


In order to prove (4.12) we set V = 1®Fys(71) and let ¢ denote the restriction 
of d to V. We abuse notation and write 7, for the action of M on V. Since 
im(¢) is a finite-dimensional space consisting of M finite vectors, there exists a 
finite-dimensional M invariant subspace W containing im(¢). Let T(g) denote 
the restriction of Ad(g) ® m2(g), g € M, to W. Since d is m equivariant, the 
space im(¢) is m invariant. Since M is connected and T is a finite-dimensional 
representation, it follows from Theorem 3.4 p. 356 and VI p. 511 [15] that 
im(¢) is T(M) invariant. Let To(g), 9 € M, denote the restriction of T(g) to 
im(@). Since 7 (resp. Jo) is a finite-dimensional representation, it is an analytic 
homomorphism of the Lie group M into the Lie group GL(V) (resp. GL(im(¢))) 
by VI p. 511 [15]. By (3.4.25) p.354 [15] we know m(exp(z)) = exp(dm(z)) 
and Jo({exp(z)) = exp(d7o(x)) for all x in the Lie algebra mo of M. Since 
dr, (x) = dTo(x)¢, we have $m, (exp(zx)) = To(exp(x))¢ for x € mo. Since M is 
connected we conclude that $71(g) = To(g)¢ for all g € M. Thus ¢ intertwines 
the M actions. This proves (4.12) and consequently (4.11). 


(4.13) PROPOSITION. Let X € t* be analytically integral and A*(€) dominant. 
Let Fys(A;) denote the representation space for the irreducible representation of 
M with highest weight A; given in (4.9). Put 


C&™) = U0) ®upy Fu (Ay) for 1 <j <p and 
cf = U(e) @upe Fu). 
There exist (£,M) module maps d},1 <j <p, ande such that the sequence 


(4.14) da Ct SS COM) SHO) 0 
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is exact. Here € is the canonical (€,M) module epimorphism. 


PROOF. By (4.7), the (@, 4) modules (ugar 0 <j <p, are just the @°* mod- 
ules C; in (4.6) on which an action of Z(@) and an action of M have been defined. 
We need only show that the ¢** module maps given in (4.4) automatically extend 
to (€, M) module maps. We first observe that on eo the action of z € Z(é) is 
by the scalar \;(z) = (op--- Op—j41(A + pe) — pe)(z). Since op-+-op—j41 © We, 
we have by (4.8) that A;(z) = A(z). Thus the action of z € Z(@) on each Gane 
0 < j <p, is by the same scalar \(z). Since ¢ = £55 @ Z(é) we conclude that 
each @°° map in (4.4) is an ¢ map. Finally, each ¢** map in (4.4) is an M map 
by (4.10). This completes the proof. 


Recall our choice of parabolic subalgebras q = @@u and q; = M@w, of g 
and the parabolic subalgebra Py = m @ ug of @. If W is an (€,M) module, we 
define u to act trivially on W and thus make W a (q,M) module. We make 
U(g) @uiq) W a (g,M) module by letting M act by the tensor action Ad @ tw 
where wy is the representation of M on W. In particular, if 7) is an irreducible 
representation of L on the space F(A) with highest weight A, we set 


(4.15) N,(A) = U(g) @u(q) Fi (A): 


In a similar way, if V is a (m, M) module we let u, act trivially on V and obtain 
the (g,M) module U(g) ®u(q,) V- If 7, is an irreducible representation of M 
with highest weight 7 and representation space Fy,(r), we set 


(4.16) Nai(T) = U(9) ®u(q1) Fu (7). 


(4.17) PROPOSITION. Let Fys(rT) be the representation space of an irreducible 
representation of M. Then N,,(r) = U(g) @uiq) Ns(t) as (g,M) modules. 


Proor. By (4.7) we know Ns(r) = U(¢)@up,) Fau(r) as (€, M) modules. By 
definition the action of u on Ngs(r) is trivial. Now by (4.1) one sees that u) = u@® 
us so that Pe@u = MOus@u = m@u, = qi. We define u to act trivially on Fy;(7) 
and consider the (q, M) module U(q) @u(p,@u) Fu(T) = U(q) @u(q,) Fu (7). We 
observe that as (¢, M) modules, we have U(L)@u pr) Fy (r) = U(q)@u(q,) Fu (7). 
Moreover, since [é,u] C u, it follows that the action of u on U(q) ®u(q,) Fiu(7T) 
is trivial. Since u already acts trivially on U(€) @u;p,) Fu(7) (by definition). it 
follows that U(£) ®u(p,) Fu (tT) = U(q) ®u(q,) Fu (r) as (q,M) modules. But 
then we have (g, /) module isomorphisms 


U(g) @u¢q) Ns(r) = U(g) @u¢q) (U(E) Gur) Fa (7) 


= U(g) @ucq (U4) ®u(qi) Fu (7) 
=U(g) @u (a1) Fru (7) = Ng, (7). 
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(4.18) Proposition. Let the weight be analytically integral and A*(é) 
dominant. Then there exists an exact sequence of (g, M) modules 


0- Ng, (Ap) > Na, (Ap-1) Ss Ng, (A) = N,(A) — 0. 


Proor. Let V be a (q, M@) module and set F(V) = U(g) @uq) V- By (6.1.6) 
[p.302, 18], F is an exact covariant functor. Define u to act trivially on the (€, M) 
modules Chae 0<j <p, in (4.13). By applying F to the exact sequence (4.14) 
we obtain an exact sequence of (g, //) modules. The first p+1 nontrivial modules 
in this exact sequence are 


F(CS") = U(g) @u(qy (U(0) Bure) Fu (ds)) 
=Nq(A;), O55 pz, 


by (4.17). The last nontrivial module is F(F,(A)) = U(g) @uq) F(A) = Nq(A) 
by (4.15). This completes the proof. 


Let o denote the conjugate linear antiautomorphism of U(g) induced by the 
map x — —z on the real form go of g. Let E be an (a,M) module. Let 
Home(E, C) denote the space of conjugate linear maps y: E — C. We define a 
o(a) = 4 action and M action on Home(E, C) by setting 


(x- p)(e) = p(a(z)e), zéa,e€ E, 
(g-v)(e)=p(g'-e), gE M,eEE. 


Let Home(E,C)yw = {y € Home(E,C) : dim(spanc{g: yp : g € M}) < co} 
denote the space of M finite vectors. As in 6.1.20 [18], the functor E — 
Home(E,C)y from C(a, M) to C(a, M) is contravariant and exact. 

Let W be a (q1, M) module and let Homyg,)(U(g), W) denote the space of 
C linear maps y : U(g) — W that satisfy yp(zy) = x- y(y) for all x € U(q,) and 
y € U(g). We define an action of g and M on Homyg,)(U(g), W) by setting 


(z-y)(y)=plyz),  2,y€ U(g), 
(9-y)(y) =9-p(Ad(g")y), 9 € M,y € U(g). 


We follow the notation of 6.1.21 [18] and let pros iy (W) = Homyg,)(U(g), W) a 
be the space of M finite vectors in the space Homyg,)(U(g), W). Then by 6.1.22 
[18], the functor pros, : C(G1,M) — C(g,M) is covariant, exact and takes 
injectives to injectives. 

Let Fy be acomplex, finite-dimensional representation space for a representa- 
tion 7 of M. Using the derived representation dz, the space Fy is an m module. 
We may view Fy both as a q; module and a q; module by defining the action 
of u, and i, to be trivial. Since M is compact, there exists an inner product 
<-+,-> on Fry with < g-e,e’ >=<e,g!-e' >and <2-e,e >=< €,0(z)e’ > 
for all z € U(m), g € M and e,e’ € Fy. But since both u, and i, act trivially 
on Fas, we have < x-e,e’ >=< e,a(x)e’ > for all c € U(q) as well. 
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(4.19) LEMMA. Let Fy, be a complez finite-dimensional representation space 
for a representation of M. View Fy as a (qi,M) module and as a (q1,M) 
module by letting both u, and q, act trivially. Then there exists a (G1, M) module 
isomorphism Fy = Home(Fu,C). 


ProorF. Define the C linear map Fy — Homeg(Fy,C) by e +> ye where 
p(e’) =< e,e’ >. This map is injective by the nondegeneracy of < -,- > and 
surjective since dim(F'y,) < oo. Moreover, by the invariance of < -,- > this map 
is a (q1, M@) module isomorphism. 


(4.20) PROPOSITION. Let Fy be a complez finite-dimensional representation 
space for a representation of M. View Fy as a (qi, M) and as a (q,, M) mod- 
ule by letting both u, and % act trivially. Then there exists a (g,M) module 
isomorphism 

prog ar (Fia) = Home (U(g) ®u(q.) Fs) 


Proor. By a standard middle linear argument, there exists a C linear map 
® : Homyig,)(U(g), Home(F, C)) + Home(U(g) @u(q,) Fu, C) such that 


(Sp)(x @ e) = plo(z))(e), 


for r@e € U(g)®Fy and y € Homyg,)(U(g), Home(Fy,C)). We now show that 
® is a (g, M) map. Let y € Homy,g,)(U(g), Home(F,C)), x,y € U(g) and e € 
Fy. Then ®(x - y)(y® e) = (x- y)(a(y))(e) = v(a(y)z)(e) = v(a(o(z)y))(e) = 
(®(~p))(o(z)y ® e) = (x- B(y))(y @ ee). Thus, O(x- y) = x- B(y) for all c € 
U(g). Let g € M. Then we ade (9 - ee = (9- v)(o(y))(e) = (9: 
y(Ad(g~")a(y)))(e) = p(a(Ad(g7!)y) (97! - e) = (®p)(Ad(g7*)y ® g7* +e) = 
(g: Py) (y@e). Thus, ®(g-y) =g- Pid ) for all g € M so that ® is a (g, M) map. 
Define VW : Home(U(g) @u(q,) Fu, C) > Homyg,)(U(g), Home(F', C)) by 


(U(x)(z))(e) = x(o(x) @ €) 


for x € Home(U(g) @u(q,) Fu, C), x € U(g) and e € Fy. Observe that the map 
e++ (W(x)(x))(e) is conjugate linear. Moreover, if g € U(q1), then Y(x)(qz)(e) 
= x(o(qz) ® e) = x(o(z) @ o(g)e) = (U(x) (x))(o(g)e) = (4: YOO (2))(e). Thus 
V(x)(qxz)=4-¥ (x)(z) so we conclude that (x) €Homyg,)(U(g), Home (Fu, C)). 
It is readily checked that W is the inverse of ®. Since M finite vectors are clearly 
mapped to M finite vectors, there exists a (g, M@) module isomorphism 


Homyq,)(U(g), Home (Fa, C)) zw = Home(U(g) @u(q,) Fu, C) ae 
By (4.19), we have Home(Fy,C) = Fy so that the proposition now follows. 


(4.21) Remarks. (1) If Fy, is a complex, finite-dimensional representation 
of L, then by restricting the Z action to M and letting u and u act trivially on 
F,, we may argue as above to conclude 


pro® i" (F,) = Homg(U(g) @uiq) Fi,C) ar 
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(2) Let W and V be two (g,M) modules. We say W is conjugate dual to 
V if there exists a nondegenerate sesquilinear pairing < -,- > of V and W such 
that <g-v,w >=<0,g°!-w >and < z-v,w >=< v,0(x)w > forallu eV, 
w€ W, x € U(g) and g € M. The proof of (4.20) then shows that proe., (Fu) 
is conjugate dual to U(g) @uiq,) Fu. The appropriate pairing in this case is 
< 9,2 @e >=< y(a(x)),e >, where < -,- > on the right side denotes the 
invariant inner product on F4,. 


(4.22) PROPOSITION. Let the weight A be analytically integral and At (é) 
dominant. There exists an exact sequence of (g,M) modules 


0 — pro?y;(Fr(A)) > prog), (F(A) > +> > pro), (Fa (Ap) > 0. 


Proor. Apply the contravariant exact functor Home(-,C)a to the exact 
sequence in (4.18) to obtain an exact sequence whose last p+1 nontrivial modules 
are, by (4.20), 


Home(Nq,(Aj),C)ar = pros (Fu(Aj)), 055 <P, 
and whose first nontrivial module is pro's; (Fr (A)) by (4.21.1). 


The resolution in (4.21) is called the BGG resolution of pros’ (F(A )). We 
now state (3.2) in the case where the module A is the (g, M) module prof’, M(FL(A)), 
the resolution of A is the resolution given in (4.22) and the functor [ = Te is 
the Zuckerman functor. We refer the reader to section 6.2 of [18] for the precise 
definition of [ and some relevant properties. In particular, by (6.2.10) [18] T 
is left exact and the category C(g,M) has enough injectives so that the right 
derived functors I’, > 0, are well defined (cf. (6.2.11) [18]). 


(4.23) THEOREM. Let the weight be analytically integral and A*+(£) domi- 
nant and let F(A) denote the representation space for the irreducible represen- 
tation of L with highest weight X. Let N denote the deleted complex 


0 — proo™, (F(A) > «++ prog), (Fu (Ap)) > 0. 
Then there exists a eS quadrant spectral sequence 
Ep” = H"(0™(N)) => I" (prof (Fr(A))) . 


As a first step in determining the E2 terms in (4.23), we now compute 
r™(N). (BY this we mean that we now identify each of the (g, A) modules 
T™ (pros, u(F'u(A;))) for all 0 < j,m where Ao = . Recall the parabolic 
q(wp) = te p, of g. We put Py, = q(wp41) Mt. Then P; is a parabolic subalge- 
bra of €. 


(4.24) PROPOSITION. If V is a (t,M) module and W is an (m, M) module, 
then one has 


(1) pros (TE nV) = i a te (pro 


os 


q(wp 
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(2) prow ay (pro (W)) = = prof") (W). 

PRoor. The first equation is a special case of (a) in (6.3.9) [18] where we have 
replaced q! and q? (in the notation of (6.3.6)) with 9, and q(w,), respectively. 
This results in L' K = M,L?0K = K and q° = P,. The second equation 
follows from (6.3.8) [18] with these same opposite parabolics. The verification of 
this, which left to the reader, is somewhat simpler since no tensoring with D, D! 
or D? is involved. 

Let W, denote the subgroup of W generated by the simple reflections {o; : 
i #p}. Then W, is the Weyl group for K. Let p, denote half the sum of positive 
roots At(t) of t. A weight 7 is said to be & singular if 7 + p; is orthogonal to 
some root a € A(t). If 7 is § nonsingular it follows that there exists 0 € W, such 
that o(7 + px) lies in the positive Weyl chamber, i.e., (o(7 + px), a) > 0 for all 
a € At(t). Moreover, the element o € W, satisfying this property is unique. 

Let 7 € t* be analytically integral and At () dominant. Since K is a compact 
linear connected reductive group, there exists a representation of K on a space 
Fx(r) with highest weight +. Suppose 7 is analytically integral and € nonsingu- 
lar. By replacing We and @ by W; and € in (4.8), it follows that o(7 + px) — px is 
analytically integral for all a € W,. If a € W, is chosen to be the unique element 
for which o(7 + px) — px is A*(€) dominant, then the K type Fx(o(7 + px) — pk) 
is well defined. 

We now state the Bott-Borel-Weil theorem for the compact group K in a 
manner convenient for our purposes. The proof appears in [10]. The version 
given here is the conjugate dual version of the theorem given in (6.2) [9]. One 
applies (6.3) [9] to (6.2) [9] to obtain the version below. The version of the 
theorem for the group U(n) in (7.35) [12] is stated in terms of opposite parabolics 
and is comparable in form to the version given here. 





(4.25) PROPOSITION. Suppose T is analytically integral and At(m) domi- 
nant. Let Fx,(r) denote the representation space of the irreducible representation 
of M with highest weight rT. 


(1) If r is singular, then (Ey aE ‘(pro whu(r)) ts zero for all j > 0. 

(2) Let 7 be & nonsingular and leto € Wy. denote the unique element for 
which o(7 + pr) — pr is At(t) dominant. Then 

i Fr(o(t+px)— pr) if 7 = eo), 


&K yj tM 
(Pe'n)” (pros yy (Fu(7))) = otherwise . 


(4.26) PROPOSITION. Suppose 7 is analytically integral and A*(m) domi- 
nant. Let Fx4(7) denote the representation space of the irreducible representation 
of M with highest weight r. 


(1) If 7 is € singular, then I? (prof M1 (Fu (7))) is zero for all j > 0. 
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(2) Let r be € nonsingular and let 0 © W, denote the unique element for 
which o(7 + px) — pr is At(t) dominant. Then 
prose (F(a o(r + pe) — pr)) if j = £(0), 


0 otherwise. 


1? (prof "4, (Fiu(r))) “| 


Proor. Recall that the functor pro is covariant, exact and carries injectives 
to injectives. We then have 


14 (pro&, (Fy(r)))= (Pea (Pe (Eu (r ))) by (4.24.2) 
= (To prot ae Ca ak (pro; OB wy (EM (T r))) since pro is exact 
=(» roe ke Pe): (pro, (Fu (tr ))) by (4.24.1) 

g,K 


= profs. e(rey ‘(prot (Fu (4 ))) since pro is exact. 


q(wp),K 
If 7 is € singular, then this last line is zero by (4.25.1). This proves (1). If 7 is 
t Bone eNee and o € W, is chosen as in (2), then by (4.25.2) this last line is 


prof Pasa o(r + px) — px)) if 7 = (0) and 0 otherwise. This proves (2). 


5. Computation of EF. terms 


For certain highest weights A, the HE. terms of the spectral sequence in (4.23) 
will now be determined. The weights A; in (4.9) are first written in a form 
which allows us to easily identify which of these are § nonsingular. For the t 
nonsingular A;, the explicit form of the At() dominant weights o(A; + px) — pr 
is then obtained. By plotting these At (t) dominant weights we obtain a table 
whose m** row may be identified with the complex [’"(N) where N is given in 
(4.23). Since the complex ['’"(N) is known explicitly by (4.26), the desired EF» 
terms result by taking cohomology of the rows. In addition, certain higher order 
terms of this spectral sequence play a crucial role in what we do in section six 
so we proceed to determine these terms as well. We begin with some routine 
calculations. 

Recall that o; is Weyl group element corresponding to the simple root a;,1 < 
i <n-—1. We adopt the convention that if an index j occurs for which w; is 
undefined, then that term is omitted from the expression. 


(5.1) LEMMA. Let A = - ajw;, wherea; € R. Lets > 1 andt > 0 be 
integers satisfying s+t<n-—1 and set o =0544---o5. We have 


(1) 


s+t—1 s+t s+t n—-1 
>» ajw; + Qi41Wi — (> Qi )Ws+t + ( 5 Qi )Ws+t41 ae 5 AjWi, 
t=s—-1l i=s i=s t=stt+l 
(2) 
s+t—-1 st+t s+t 


o(A)—-A = adgws_1 tn ys 0541-4; )w; —(as4e+)_ ai) weet (>- as)wepett- 


j=s i=s i=s 
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Proor. Using the fact o;(w,;) = w; — 6;;a;, it readily follows by induction on 
t that o(A) has the form given in (1). The expression for o(A) — A in (2) then 
follows. 

(5.2) DEFINITION. Let A = an a;w; and for each integer j satisfying 1 < 
j <p set Ay = Op- ++ Tp—741(A + p) — p. We also put Ao = A. 

Since p — pe is a multiple of w,,; and since each product o,---op_j+1 fixes 


Wp41, it follows that A; = Op---Op—j4i1(A + pe) — pe. Thus, the form of the 
weights A, defined in (5.2) agree with the weights defined in (4.9). 


(5.3) LEMMA. For each integer j satisfying 1 <j <p the weight r; has the 
form 


p-j p-l P 
Aj = Wp-j + So aj + S- i410; — ( > at+jt+1)u, 
i=l i=p-j i=p—j+l 
p+ n-1 
+ ( S- ai + j)Wp41 + S- QjW};. 
i=p—j+ i=p+2 


PRooF. With s = p—j+1 and t = j — 1, it follows by (5.1.2) that 
Op* + Op—j41(P) — P = Wp_y — (F + lwp + jwp41 and by (5.1.1) that 


p-j p-1 P 
Op ++ Fp—541(A) = Se aww + S> Qji410i — ( Ss Qj Wy 
i=1 i=p-j i=p—j+l 
P n-1 
+ ( > A; )Wp41 tt Ss QjiW;. 
i=p—jt+l i=p+1 


The lemma follows by adding these and combining coefficients. 


(5.4) DEFINITION. If A = is a,w; and »4;, 0 < j < p, is defined as in 
(5.2), then for each j € {0,... ,p} and v € {1,... ,qg—1} define A, = AZ (A) by 
Al = —(Aj + Pk, @p41 +-++ + Op4v)- By (5.3) we have 

pty 
Al =-v—j- S> Qj. 
t=p—jtl 

Motivated by (4.26), we now determine the € singularity of the weights 4, 
that appear in the complex N in (4.23). Recall that the weight A = Say QjW; 
is integral if and only if a; € Z for i € {1,... ,n2 —1} and is dominant for At (2) 
if and only if a; > 0 fori Ap+1. 


(5.5) PROPOSITION. Assume A = PSS, aw; is integral and At (£) dominant. 


(1) The functions j ++ A}, and v + A? are strictly decreasing Z valued 
functions. 

(2) The weight »; is t singular if and only if there exists av € {1,... ,qg—1} 
such that A) =0. The value of v with this property is unique. 
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(3) The weight ; is At(t) dominant if and only if Ai <0. 

(4) If A; is & nonsingular and not At(t) dominant, then there exists v € 
{1,... ,qg—1} such that AZ > 0. Define v; to be the greatest such integer. 
Then Op4v,°**Fp41(Aj + pe) — pr is At(€) dominant. 


PROOF. Part (1) is clear by the explicit form of A? given in (5.4) and the 
assumption that A is integral and At(é) dominant. Since A is A*(é) dominant, 
Gp+1 is the only possible negative coefficient in A. By the form of A; given 
n (5.3), it is clear that (A; + px,@) = 0 for some a € At(€) if and only if 
O@ = Op41 +--+ +Qp4y for some v € {1,... ,q—1}. If such a v exists, it must be 
unique by (1). This proves (2). 

To prove (3), note by (5.3) that A; is A+ (€) dominant if and only if (A;,@p41) 
> 0. But A? = —(Aj + px, %p41) so that Aj < 0 if and only if (A;,a@,41) > 
—(px, p41) = —1. Since A is integral, so is A; by (5.3). Consequently, (A;,@p41) 
is an integer. This proves (3). 

Now suppose 4, is § nonsingular and not A*(t) dominant. By (2) we know 
that A? #0 for all v € {1,...,q—1}. By (3), we must have Aj > 0 so that v; 
in (4) is well defined. Then v; satisfies Al. > 0 and, ifv; <q-1, Aba <0 by 
(1). Since each o; with i # p fixes wp, we have 


Opty; ++ Opa (Aj + px) — pk = Opty; = Opa (Aj +p) —p 


= Opty; *** Oppi(Op* + Op—j4i(A + p)) — p 


by the definition of A;. By (5.1.1) we have 


ptv;—1 pty; 
Opipy 0 pgi4ae = ajw; + ; Qj 41W; — y Qi )Wp+v, 
i=p-j an 
ptyj+1 n-1 
+ ( ) Qj )Wp4v, 41 + ) AW; 
1=p—jtl i=ptvj;+2 


and by (5.1.2) we have op4v, °*-Op—j41(e) — 9 = Wp_j — (Vj +7 + lwpsr, + (Yj t+ 
j)Wp+y,4+1- If vj < q—1, then by (5.4) one has 


ptv;-1 
Opty," Opal; + Pr) — Pk = Wp_j + : aw + 5 Aji 41% 
i=p-j 
n-1 
+ (Ai. — lwptr, — (Ad 41 + Ljwpty;41 + 5 Ajwi. 
t=ptv;42 


Since both (Al, — 1) and —(A}, 41 + 1) are nonnegative integers, it follows that 
Opty, °° p41 (Aj + Pe) — pe is A*(t) dominant. The v; = q — 1 case is similar 
and its proof is omitted. This completes the proof of (4). 
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(5.6) COROLLARY. Assume X is integral and At (£) dominant. 
(1) If Als > 0 for some j € {0,...,p}, then each r, with r < j is 
nonsingular and not At (t) dominant. Moreover, the integer v, defined 
in (5.5.4) is q—1. 
(2) If there exists v’ € {1,... ,q—2} satisfying A?, > 0 and Al iss < 0, then 
A; is € nonsingular and not At (t) dominant. Moreover, v; in (5.5.4) is 


v’. 


PROOF. Suppose Aly > 0 for some j € {0,... ,p}. From (5.5.1), we know 
i es Aly for all r < j. Thus, Aj_, > 0 so by (5.5.2) we conclude 4, is € 
nonsingular. We also know 4,,7 < j, is not At (#) dominant by (5.5.3). Finally, 
by the definition of v, it is clear that v, = q—1. This proves (1). 

Suppose A}, > 0 and Als < 0 for some v’ € {1,...,q— 2}. By (5.5.1) 
it follows that A? is never zero. Hence A; is & nonsingular by (5.5.2) and is 
not At (t) dominant by (5.5.3). Finally, by the definition of v;, it is clear that 
v; =v’. This establishes (2) and so completes the proof. 


We now introduce notation used to label the reduction points in the En- 
right, Howe and Wallach classification of unitary highest weight representations 
of SU(p + 1,q—1) [8]. We are interested in the reductions points since these 
weights are precisely the highest weights in the classsification for which the pro- 
duced module prof’, (F(A)) resolved in (4.22) is reducible. The reducibility of 
pro8't7 (F(A) is essential here for otherwise (by Theorem (6.3.5) and Proposi- 
tion (6.4.6) [19]) the modules I? (pro®'y, (F,(A)) are zero for 7 4 S, the middle 
dimension. Since we wish to study the derived functors I for 7 < S, we naturally 
begin with reducible modules. 

By tables (6.21) and (6.22) [5], the reduction points for SU(p + 1,q— 1) are 
A*(é) dominant integral weights of the form 


, 


Pp n—q 
(5.7) A= So awi-(n-p'-q +€-14 So ai)wpy1 + ye Gs 
t=p! ixptl i=p+2 
Here the integer p’ is defined as follows: if there exists an index i in {1,... , p} 


such that a; # 0, then p’ is defined to be the smallest such integer. If a; = 0 
for all i € {1,...,p} we set p’ = p+ 1. Similarly, if there exists an index 
i € {l,...,q@— 2} such that a,_; 4 0, then q’ is defined to be the smallest 
such integer. In the case where a,_; = 0 for all i € {1,...,q — 2} we set 
q =q-—1. The integer @ > 0 is called the level of reduction of 4 and must 
satisfy 1 < @€ < min{p’,q’} in order for the A*(¢) dominant integral weight 
in (5.7) to be a reduction point. Moreover, since each reduction point is the 
highest weight of an irreducible representation of L, it is therefore analytically 
integral. Throughout the rest of this article we work only with A in (5.7) that 
are reduction points. 
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(5.8) PROPOSITION. Assume A in (5.7) is a reduction point. 
(1) IfA; is € nonsingular andv; < q—1, thenj > p—p’+é andv; < q—q'-l. 
(2) The weight r»»_p +e is & nonsingular. If q—q’ —1=0, then Ap_pze is 
At (t) dominant integral. If q—q'—1> 0, then On—q'—1- +: p41 (Ap—p' +e 
+ pr) — pr is A*(#) dominant integral. 


PrRoor. Suppose 4; is § nonsingular and v; < q— 1. By (5.6.2), we know 
AL ai ti< AL, ie, 1< Al, — Ati: = 1+ appity,. Since 0 < apyr,41 and 
Apr = 0 for r > q—q' we conclude v; < q— q’ — 1. Now consider the constraint 
Al,41+1< 0 in (5.6.2). This inequality is equivalent to 


(p—p'+@—j)+ +(q-—q'-1-»;) )+ Soa 3B a; <0. 
i=ptv;+2 


By assumption, each a; appearing in this sum is nonnegative. Since we also have 
v; <q-—4q' —1, we conclude p— p’ + £ < j. This proves (1). 
To prove (2), we first consider the case gq — q’ -1=0. Then 


Apop +t a+ Sas 3 a; =-1<0 


i=n—q'+1 


so that by (5.5.3) we know Ap_p +e is At(t) dominant. Now suppose that g — 
q’ —1>0. Then 


n-1 
p—p'+é : > + 5 = 
ea iy =, ait a; ap Qn—q' > 0 


i=n-q' 
and 
n-1 
p—p+e _ 
Ay-g = 1+ »; 
i=n-—q't+l1l 
Thus vp_p4e = @—q' — 1 by (5.6.2) 80 On_gi—1 +++ Op41(Ap—p'te + Pk) — Pk is 
A*(t) dominant by (5.5.4). This completes the proof of (2). 
We continue to determine all the values of 7 between p—p’+¢ and p for which 
A; is € singular. We therefore assume ¢ < p’. If g— q’ — 1 > 0, define the set S 
to be 


S=(s,=t+ S- a,:0<t<q-q-2 
i=n—q'—t 
Note that S$ is a strictly increasing sequence of q — q’ — 1 positive integers. 
(5.9) PROPOSITION. Let \ in (5.7) be a reduction point and assume £ < p’. 
(1) Ifq—q —1=0, then Ap_pyeys ts At(t) dominant integral for each 


s€{l,...,p’ — Q. 
(2) Ifqg—q -1>0 ands e {l,... ,p’ — } lies in S, then \p_prrers is € 
singular. 


COVARIANT DIFFERENTIAL OPERATORS 185 


(3) Letqg—q’-1>0 and 544, ES. Ifse€ {1,... ,p’ — 2} lies between s, 
and 8141, then the weight Ap_p+e+s 1s & nonsingular and On_g'-t-2°°- 
Fn+1(Ap—p’ +e+st+Pk)—pr ts At (€) dominant integral. If s € {1,...,p’— 
£} satisfies s > sg_q/-2 = max(S$), then Ap_pye45 ts At(€) dominant 
integral. 

(4) Ifg—q —1>0 ands€ {l,... ,p’— €}, then rAp_pryeys is € singular if 
and only ifs € S. 


Proor. Assume q — q’ — 1 = 0. Then for each s € {1,... ,p’ — £} we have 
Ap? tets = —s—1. By (5.5.3) we conclude that Ap_p4e4s is At (€) dominant 
integral. This proves (1). 

Suppose now that q—q’—1>0ands€ {l,... ,p’— @} lies in S. Then s = 5; 
for some t € {0,... ,qg—q’ — 2}. One has 


t 


n-q 
Apri =—(se-t- So ai). 
i=n—q’—-t 
But this is zero by the definition of s;. Hence by (5.5.2) we conclude that 
Ap—p'+é+s, is § singular. This proves (2). 
Suppose q— q’ —1 > 0 and s € {1,...,p’ — 2} with sy < s < s¢41 for 
St, 8t11 © S. We have s, +1<s < 544; —L,ie, 


, 


n—q’ n—q 
(5.10) t+1+ So a<s<t+ YO wy. 
t=n—q'—-t t=n—q'—t-1 


We find by the left side of (5.10) that 


n—q’ 
Ap~Pi+ets gg > ees 


q—q'—1-t 
i=n—q'-t 
and by the right side of (5.10) that 
n—q 
LG =t+l—s+ Se a;>1. 
i=n—q’-t-1 


Thus by (5.6.2) Ap_pye+s is & nonsingular and vp_»ieps = q—q’ —t — 2. 
Furthermore, on ~9/—t-2°** Opti (Ap—p'4e+s +P) — px is At (£) dominant integral 
by (5.5.4). This proves the first part of (3). We now suppose that s € {1,... ,p’— 
£} satisfies s > sg—q:-2 = max(S). One then finds App +tts —(8—Sg_g'-2) < 
0 so that by (5.5.3) we conclude that A,» 4245 is At (€) dominant integral. This 
completes the proof of (3). Finally, we observe that (4) follows immediatedly 
from (2) and (3). 

We summarize (5.9) by plotting the pairs (j,v) such that op4,---Op41(Aj + 
Pr) — px is A*(€) dominant integral. For brevity, put 


(5.11) J=p—-pt+flandN=q-q-1. 
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If the set B = {7 :0 < j < J—1,\A; is € nonsingular} is nonempty, we put 
jo = max(B). Observe that for any j satisfying 0 < 7 < J —1 one has Aj >0 
so that by (5.5.3) each A; with 0 < 7 < J—1 is not At(t) dominant. If A; 
with 0 < 7 < J—1 is € nonsingular, then the integer v; must be q — 1 by 
(5.8.1). Consequently, A?_, > 0 and so by (5.6.1) it follows that if B # , then 
v; = q—1 for all 0 <j < jo. We include these observations in the following tables 
by assuming B # @ and indicating jp. Table 5.12.1 (resp. 5.12.2) corresponds 
to the case N = 0 (resp. N > 0). 


Table 5.12.1 








Table 5.12.2 








q-l je e0--- 0 0 0 0 0 
N e e 
N-1 ° e 
N-1-t e@---@ 
0 0 0 0 0 0 0 
O--- jg oc: J--- J+sq --- J+s,--- Jt+ts, -°- J+St41°°° 
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Put EY’ = T3 (pro 4 (Fi (Ai)))- The EF, terms of (4.23) are obtained by com- 
puting cohomology down the rows of (5.12). More precisely, if d’ ; EY = Bee 
denotes the differential of bidegree (1,0), then one has E3” = Ker(d’)/im(d{*”). 
Similarly, the E,,, term of the spectral sequence is determined by the differen- 
tials di? : Et) — Et+"J—r+1 (of bidegree (r, 1 —1)) and is defined to be 


(5.13) BY), = Ker(di4)/im(di-"*7-"). 


We now find certain E,, terms. Set J = N+ J, where N and J are in (5.11). 
In order to count the number of columns between the J*® column and the next 
nontrivial column, set © = {s > 0: Ay+5 is § nonsingular} and define the integer 
s. € {0,...,p— J} to be 


aaa an if o£, 
F Ss, = 


0 ifL= 2. 
(5.15) PROPOSITION. Assume that I <q-1. 
(ly) fal S00 J. 
(2) If s. =0, then EVN =...= EXN. Ifs, >1, then EN =... = EDN 
and E2:N = BIN, = Ker(ds"). 
(3) Ifs, > 2, then ieee ezists a chain 


= ~N—-s.+1 »N—s.tl 
Bay Peek er dy ee Py ee Wend eat. eyes 


Cc Ker(ae ee ett) ‘a Daas era 


Proor. Assume I < q—1. It is clear by (5.12) that the only nonzero entry 
on or below the line y = —x + J occurs at the point (J, N). In particular, this 
implies that Ee ~“ = 0 if u # J from which one concludes E%!~-" = 0 ifu # J. 
This proves (1). 

To prove (2), recall by (5.13) that E% a= = Ker(d') /im(d‘-"J+"—') where the 
differential d‘-"3+"-! maps Ei-™J+r— r to E23. Since the point (J—r, N+r—1), 
r > 1, lies below the line y = —a + J, it follows that EJ~"%*""! = 0 for 
r > 1. Since E2-™N+r-! is a subquotient of E/~"%*"~', we therefore have 
m(dJ-"N+r-1) = 0 for r > 1. Consequently, £2; = Ker(d?-") for r > 1. 

We now consider the differentials di?“ : EN — EJ+™N-r+! and the spaces 
Est+rN-r+l for r > 1. Let us first consider the case where s, > 1. We claim 
that the form of (5.12) implies that there exists only one possible nontrivial 
ey REE for r > 1 and that is when r = s,. For the case N = 0, (5.12.1) 
makes this obvious. For the case N > 0, one has s,—1 zero columns following the 
J*® column in (5.12.2). The only nontrivial entry in column J+s, has coordinates 
(J + sx,N —s, +1). Consequently, we have | a ala =OQOforl<r<s,. 
We also have E3+"™%~?+! = 0 for r > s, by (5.12.2). Thus each of the maps 
dN. BIN _, BJ+nN-r+] 7d g,, is trivial since EJ+™N—T+! is a subquotient 
of BJtN-*+!. Since dN = 0 forl <r < s,, we have EY’ =... = EDN 
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and since dN = 0, for r > s,, we have Ker(d") = BY", =... = BLN. This 


proves (2) in the case where s, > 1. If s, = 0, then by definition of s,, there are 
only zero columns following the J‘® column. Thus d?% = 0 for r > 0 so that 
E7N =...=E2N. This completes the proof of (2). 

To prove (3), we assume s, > 2. By definition of s,, there exist s, — 1 
zero columns between column J and column J + s,. In particular, we have 
Brant = 0 for 1 <i< s,—1. Since 7 aie is defined on | Onael e a 
subquotient of Bite Ne i it follows that a les =0forl<i<s,—1. But 
then we have 
(5.16) 


Beitr Tt = Ker(dy p88") fim(dg A") = Ker(dg tp), 


for 1 <i<s,—1. Fori=1, we have EJtsN—-s-+1 = Ker(dl Sloth): But 
the map d?+*:"~*-+1 is defined on Ra ae ee Ker(d2*3:%~*-*1) so that 
Ker eer) C Kenta Sey Using (5.16), one continues in this 
way to establish (3). This completes the proof. 


(5.17) COROLLARY. Assume that I < q—1 ands, > 1. Then the differential 
di’ mops BY” to Bek 


PRooF. The case s, = 1 is clear since d7" is of bidegree (1,0). Consider 
the case s, > 1. By (5.15.2) we have Bes = BZN. By (5.15.3) we have 
EJtseN-se+1 © EitseN-s +1 Since dN maps EXN to Butt N—se+l, the 
corollary follows. 


We end this section by describing the connection between the integer s, in 
(5.14) and the weight in (5.7). 


(5.18) PROPOSITION. Let \ in (5.7) be a reduction point and suppose s, > 1. 
(1) If N =0, then s, =1. 
(2) Assume N > 0. Then s, = 1 if and only foe g >l. If@n-g =1, 
then s, < N+1 and s, is given by 


opel ifQ4g, 
8, = 

N+1 if Q- 9, 
where Q = {t:1<i< N-1 and an_g_i > 0}. 


Proor. If N = 0, then by (5.9.1) we know A741 is At(€) dominant integral 
and is € nonsingular. By the definition of s, we conclude s, = 1. This proves 
(1). 

Now assume N > 0. By (5.9.4), the first zero column following the J‘® column 
is J+ sq. Therefore s, = 1 if and only if so > 1. But so = a,_,’. This proves 
the first part of (2). 

Now suppose that an_g’ = 1. Recall that columns J+1,...,J +s, —1 are 
zero. By (5.9.4) the zero columns to the right of column J are determined by 
the set S = {s,:0<t< N—1}. Consequently, s, —1< N and s; =t+1 for 
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O<t<s,-2. Since an—g = 1 and s, =t+ 0%” a; forO0 <t<N-1, 


i=n—q’—-t 
one has > Suara a; = 0 for0 <t<s,—2. Since a; > 0 fori #p+1, we have 


(5.19) aj =O forn-—q’-s.+2<i<n-qd-1. 


It follows by (5.19) that s, = N+1 iff Q = ©. To prove (2), suppose Q # 
@. Then s, < N and by the above discussion one has S = {1,2,...,S. — 
1, 8(s,-1))--- »8n—1}- Since column J+s, is nonzero, s, ¢ S so that sx < 8(5,-1)- 
By (5.19) this inequality is equivalent to an—q/—s,41 > 0. Thus s, — 1 = min(Q). 
This completes the proof of (2). 


6. Identifying covariant differential operators 


The purpose of the present section is to show that the differential d2- in 
(5.15) is a multiple of a distinguished covariant differential operator (CDO). In 
general, CDOs are G invariant differential operators defined on representation 
spaces of holomorphically induced representations of a Hermitian symmetric pair 
(G, K). We refer to [6] for the explicit form of such operators in standard realiza- 
tions of the induced representation spaces. By [6], a special property of certain 
nonzero CDOs is that the € finite vectors in their kernels form irreducible unita- 
rizable (g, K) modules. By utilizing this property, cohomology can be realized 
as irreducible unitarizable (g, K) modules. We begin with a brief discussion of 
CDOs. 

Suppose U, and U,, are holomorphically induced representations of a Hermit- 
ian symmetric pair (G, K). A covariant differential operator D,,,, is a continuous 
operator intertwining U, and U,,. These operators arise as dual maps to partic- 
ular g maps between generalized Verma modules. It is known from the work of 
Boe, Collingwood, Enright and Shelton (cf. [2], [3]) that the spaces of such g 
maps are at most one-dimensional. By duality, the same is true of CDOs. Thus 
a CDO is determined, up to a constant multiple, by the inducing parameters T 
and pL. 

We are interested only in those CDOs that realize irreducible unitary (g, K) 
modules as their kernels. Such CDOs arise in the context of unitary highest 
weight representations. More precisely, if 7 is a reduction point in the Enright, 
Howe and Wallach classification [8] and if 4 = u(r) is the highest weight of the 
maximal submodule of U(g) @u(t@p,) F(T), then the space of K finite vectors 
in Ker(D,,,,) is an irreducible unitary highest weight (g, A) module (cf. (3.2) 
[6]). All pairs of weights (7,4) for which D,,,, has this property can be found 
in section six of [5]. For convenience we give such relevant pairs below. Before 
doing so, we recall the cone decomposition of the set A, of reduction points. 
By (6.6) [5], A, is the disjoint union of finitely many cones. A cone C in this 
decomposition is defined by the condition that 7 — u(r) is constant for all 7 in 
C. In the SU(p,q) case, each cone can be parametrized by a triple (a, b,m) of 
positive integers. Here a and } measure a kind of singularity in the weight r 
and m is the degree of the CDO D,,,, . We summarize the results of (6.21) and 
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(6.22) in [5] for the group SU(p, q) and give the relevant CDO pairs (7, 4) along 
with the associated cone data (a,b, m). Recall that n = p+ q. 


(6.1) DEFINITION. We say that (7, 4) is a CDO pair (for SU(p, g)) with cone 
data (a,b,m) if: 7 and yz are At (€) dominant integral weights of the form 


p-1 n—b 


r= Drews - (n-a-b+m—=14 Sei)wy + > Ci, 


i=a ixp i=pt1 


and 


p-l 
[= We—-m + (Ca — 1)Wa + » iw, — (n-a-b+m-1+)oci)wy 


i=at+l ifp 
n—b-1 
+ 5 ChW + (Cuce =~ 1)Wn—s + Wn—b+m 
i=p+1 


where a € {1,... ,p} and b€ {1,... ,q} are defined by 
(eo sp—-l},q>0} ifdie {1,...,p—1} with ¢ > 0, 
a= 


Dp if c) = --- =cp_, =O, 
j fa er tee ten, if die {1,...,q—1} with c,_; > 0, 
q if Cpp1 = ++ = Cy_1 = O, 


and m is an integer satisfying 1 < m < min{a,b}. In the case a = p, then the 
term (cg — 1)w, appearing in ys is replaced with —w, and similarly, if b = q, then 
the term (cn—» — 1)wp—» in ps is replaced with —wy. 


We resume the notation of section five. In particular, recall the form of A in 
(5.7), the definition of the integer s, in (5.14) and integers J and N in (5.11). 
Note that if s, > 1 then ¢ <p’ and by (5.18) we have s, < N+1. 


(6.2) PROPOSITION. Let X in (5.7) be a reduction point and assume s, > 1. 
Put 


a if N =0, 
T= 
Op+N °° Oper (Ay + pr) — Pk if N > 0, 
Ast. if N = 0, 
b= AJ+N41 if N>0 ands, =N+1, 


Opt+N—s.tl*? “Opt (Ar4+s, +pr)—pr ifN>Oands,<N+1. 
Then (rT, ) is a CDO pair with cone data (p’ — £,q' + Sx, Sx). 
ProoF. First consider the case N = 0. By (5.18.1) we have s, = 1. Now by 


(5.8.2) A; is At+(#) dominant integral and by (5.9.1) Ay41 is At(€) dominant 
integral. Moreover, by (5.3) we have 


p-l 


P 
Ag = Wye + S- Qj41W; — (J+1+ > ai) wp 


i=p’—£ i=p!—f+1 
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and 


p-1 p 
Arr =wpeit SP ainwi-(J+2+ D7) ai)uy tuper. 
i=p’—£-1 ix=p’—£ 


Now ay_¢ = 0 so we see by (6.1) that (Aj, 4.741) is a CDO pair with cone data 
(p' — £,q,1). Since N = 0, we have g = qg' +1 = q’ + ,. This proves the 
proposition for N = 0. 

Now consider the case N > 0. By (5.8.2) we know that tT = op4n-°++Op4i(Au+ 
Pr) — Px is A*(t) dominant integral. By the proof of (5.5.4) with vy = N we 
have 


ptN-1 

(6.3) = Wy’ —¢ oo oa Qj4+1W; + (Az = 1)wWp+Nn = (Adios ob lind wees 
i=p!’—£ 

By the proof of (5.8.2) we know Af, = a,-g and Af,, = —1. In order to 


identify the precise form of the weight 7 in (6.3), we must consider the cases 
Qn—q > 1 and a,_q = 1 separately. 

Suppose first that an_g’ > 1. Then we have T = wy_¢ + Rar Qj41W;i + 
(dn—q’ — 1)wp+n. Now by (5.18.2) we have s, = 1. Set = opin ++: Opsi(Asqit 
Pr) — Pk. By the proof of (5.5.4) we know py takes the form 


p+N=1 
= Wye + > Qj410j + (Ade a l)wp+n _ (Ant + 1)wp4Nn 41: 
i=p!—f-1 
But Ay*? = an—q — 1 and Aji) = —2. We see that y is At(#) dominant 


integral. Moreover, by (6.1) it follows that (7,4) is a CDO pair with cone data 
(p’ — £,q'+1,1) = (p' — £,q' + 5,4, 8%). This proves the proposition in the case 
Aan—qi > 1. 

We now suppose an_q = 1. By (5.18.2) we have 1 < s, < N+1. According 
to (5.18.2), we need to consider the cases s, = N+1 and s, < N+1 separately. 
We assume first that s, = N+ 1. Then the set Q in (5.18.2) is empty so that 
Qp+2 = °+* = Gn—q/-1 = 0. Then 7 in (6.3) is 


Pp Pp 
T = Wye + S Qi 41Wi = Wyre + S Qj 413. 
i=p!—£ i=p’—1 


By (5.3) we know 4 = A74N41 has the form 
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p-l Pp 


Wy—(J+N-+1) + :y Qj41Wi — (J+ N+2+ a a: up 
i=p—(J+N+1) i=p—(J+N) 
ptl p-l 
+ (J+ N+1+ sy a; )wp1 + Wy—q! = Wp—e—s, + x. Oi41Wi 
i=p—(J+N) i=p’—-1 
Pp ptl 
= (J+N+2+ dai) uy + (J+N+ 1+ DY ai)wpas + Wn—q’: 

i=p’ i=p! 


Since —ap41 =J+N+1 eee a; and p+s, =n-—4/, it now follows by (6.1) 
that (7, 2) is a CDO pair with cone data (p’ — ¢,q, 8.) = (p’ — £,q' + Sx, 8x). 

We now consider the case a,_,/ = 1 and 1 < s, < N+1. Then by (5.18.2) 
we have 0 = @,-9/-1 = +++ = Gn—q’—s. +2 and Qn_q/-s,41 > 0. From (6.3) we see 
T = Op4n-**Op4i(As + Pe) — Pe has the form 


ptN-1 n—q'—S. 
T= Wet y Qi41Wi = Wyre + S Qi 41W3. 
i=p’—é i=p’-1 


With p = op4n—s.41°°*Op4i(As+s. + Pk) — Pk, We know by the proof of (5.5.4) 
that y has the explicit form 
n—q'—s.—1 


J+S.« 
Wp! —l—s. oF we Qj41Wi + (An a4 = 1)Wn—q’—s. 


i=p’-1 
J+S« 
— (An 349 + Dwn—g—s.t1 + Wn—¢- 
One finds that Ayt*? ,, =@n—q'—s.41 and Ajt’: |, = —1. It now follows from 


(6.1) that (7,2) is a CDO pair with cone data (p’ — é,q'+.,8,.). This completes 
the proof. 


We now make clear the connection between the produced modules appearing 
in (4.26), the differential dz:" in (5.17) and the CDO pairs (r, 11) of this section. 
We begin by recalling a well-known fact regarding produced modules. Let 7 
denote an analytically integral, A*+(€) dominant weight and let Fx. (7) denote 
the representation space for the representation of K with highest weight 7. A 
standard realization for the holomorphically induced representation space corre- 
sponding to 7 is the space O(r) of Fx (1) valued holomorphic functions on G/K. 
The representation of G on this space is given by a multiplier representation (cf. 
(1.2) [5]). By (1.10) [5] the K finite vectors in O(r) form a (g, A’) module equiv- 
alent to pros 5 x (Fx (7))- Consequently, by restricting to the space of K finite 
vectors, we may view the space of CDOs as a one-dimensional space P(r, z) of 
(g, K) module maps 





. gk g.K 
Dry: prof (P(r) > prot (Fes) 
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where (7, 4) is a CDO pair. 
Recall from section five that J = J+.N where J = p—p’—£and N =q-q’-1. 


(6.4) THEOREM. Let X in (5.7) be a reduction point. Assume s, > 1 and 
I<q-l. 


(1) There exists o € W with £(0) =I such that 


Ker(d2:%) = 1°) (pro8s7 (Fi (A)))- 


(2) The differential d/N is a multiple of the CDO D,,, whose parameters 
are given by the CDO pairs (r, 1) in (6.2). 

(3) If d2% is nonzero, 1“) (pro®7 (FL (A))) is an irreducible, unitarizable 
highest weight (g, K) module with highest weight r = 0 (A+ p) — p. 


PROOF. We begin by proving (2). Assume A is a reduction point and s, > 
1. Then & < p’ and by (5.9) we know that (5.12) is a plotting of EB?” = 
T’(proe (Fu (Aj)), 0 < jv. If I < q—1, then by (5.17) the differential 
dJ-% maps Ey’ to EY ***%~***1_ But by (4.26) we have 


JN JK K 
Ey = prot (Fk (optw «opti (As + Pe) — Pk) = prow ,.(Fx(7)) 


and 


Byte N89" = prot (Fk (Optn—s.t1 "Opti (Aste. + Pk) — Pk) 


= pro& 





where (7, j1) is clearly the CDO pair in (6.2). Thus dz:" lies in the one-dimen- 
sional space D(r, 4). This proves (2). 
We now prove (1). Since s, > 1, we have by (5.15.2) that BY = Ker(d2). 


By (5.15.1) we also know that EX’~“ = 0 ifu# J. Since Ey” => 

r"+™ (prof (Fr(A))) by (4.23), it follows by (5.4), p.326 [4] that BZJ-J = 
I! (pro8'yy(F1.(A))). This proves that Ker(d/V) = I! (pro8'y4 (F(A). It re- 
mains to show that there exists 0 € W with (0) = J and 7t = o(A+ p) — p. 


Since p— px is a multiple of w, and since the Weyl group element op+N ---Op41 
fixes wp, it follows that 


T = Optn + Oppi(Au + Pk) — Pk = TptN°** Spt (Aj + Pp) — p- 


But by (5.3) we have Ay = op---op—y4i(A + p) — p. Consequently, by putting 
oO = Optn***Op—J+1 We then have 7 = o(A+ p)—pand &(c) = J+N =I. This 
proves (1). 

To prove (3), assume dJ:" is nonzero. By (2), this differential is a distinguished 
nonzero CDO and so by (3.2) [6] Ker(d:%) is an irreducible unitary highest 
weight (g, A) module with highest weight 7. Then (3) follows by (1). 
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(6.5) COROLLARY. With the assumptions and notation of (6.4), the (g,K) 
module T)(pro®t7(FL(A))) is irreducible if it is a proper submodule of 

gk F 
pro « EK(7)). 

Proor. If T2) (profs (F(A))) is a proper submodule, then the differential 
d2-% is not zero by (6.4.1). But then by (6.4.3), it follows that this submodule 
is irreducible. 


7. An irreducibility condition 


This section provides a sufficient condition for which the (g, K) module 
pee) (pro? tz (FL (A))) with as in (6.4) is irreducible. By (6.5), it suffices to 
show that this submodule is proper in prot< x (PK (7)}). We turn to multiplicity 

Wp), 


results to identify conditions under which the K type Fx (u) of pros= nh (Fx (r)) 


is not contained in T?) (profs (F,(A))). We begin by considering a spectral 
sequence that determines the multiplicity of K types in derived functor modules. 





(7.1) THEOREM. Let W be an (£,M) module. Choose a positive system of 
roots AP of & which is compatible with the parabolic amt. Let (7,Z) be an 
irreducible representation of K. Put S = dim(unNt) and D; = A%™@N8) (aA £). 
Then there exists a spectral sequence 


QD Homa (HS—"(aNt, Z) @ Di, S™(UN p) @ W) = Ext} y4(Z, pro¥y7(W)). 


m=0 


Here the integer r is fixed but depends on W and Z. 


PROOF. This is the spectral sequence appearing in the proof of (6.3.12) [18]. 
Here we have replaced the parabolic q by q and dropped the tensoring of W with 
D = A%™(u, Keeping in mind these minor changes, we find (6.3.20) in [18] is 
replaced by the above spectral sequence. 

By (6.2.15) [18], the multiplicity of (7, Z) in T*(pro8'y7(W)) is given by the 
dimension of Ext) y4(Z, pro’'s7(W)). Since the above E> terms determine multi- 
plicity of K types in cohomology, we proceed to write these terms in a convenient 
form. 

We now apply Kostant’s theorem ((3.2.16) [18]) to write H5—*(ant, Z) @ Dy 
as the direct sum of M types. The relevant parabolic of t is _Nt = mM (UNE) so 
that we need to choose a positive root system of & containing the roots of un &. 
We make this choice however so that the positive roots At(m) of m remain 
unchanged. Recall from section two our choice of positive roots At of g. With 
Q; = €; — &j41,1 < i < n—1, the set of simple roots corresponding to At is 
B, = {aj :1<i<n-—1}. The set of simple roots corresponding to At (€) and 
At(m) are B, = {a;:7 # p} and S(m) = {a; :i # p,p+ 1}, respectively. 

We put wo = 0p41°°'On-1 © We and consider a second choice wo(At(t)) of 
positive roots of &. Let @ (resp. @ 9) denote the length function for A*(€) (resp. 
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wo(At(€))). Let At(um€) denote the positive roots of uN €. Let px, px,9 and 
p(unt) denote half the sum of roots At (8), wo(At (€)) and At (une), respectively. 
Put 
We = {w € We: At(m) c wAt(8)} 
and 
Wao = {w EW: At(m) C wwo(At(é))}. 


(7.2) PROPOSITION. With the above notation we have 
(1) wo(A*(t)) = At (m) U(-AF (une). 

(2) pro = pk — 2p(une). 

(3) (Weo)wo = We- 

(4) lo(w) + £(wwo) = S for w € Who. 


PROOF. One readily checks that the simple roots wo(B,) for wo(At(€)) is the 

set 

wo (Br) = {01,... ,Q%p—1, Mpt2,--- »Qn—1,—(Qp41 +++ + On-1)}. 
In particular, one has S(m) C wo(B,) so that At(m) C wo(At(€)). In addition, 
we see that —A(unt) = {-(ap41 +--+ p45) 2:1 <7 < q—1} is also contained 
in wo(At(€)). This proves (1). 

To prove (2), first note that px = pm + p(uM€), where pm is half the sum of 
positive roots of m. By (1) we have px,o = Pm—p(uNk) = pmtp(une)—2p(unk) = 
Pr — 2p(uN t). The proof of (3) follows directly from the definitions of Wi and 
Who: 

To prove (4), for each w € Wy define sets Af (w) = {a € wo(At(t)) :wotae 
—wy(At(#))} and At(w) = {a € At(€) : w-!a € —At(€)}. Then by definition 
of the length function £9, o(w) = #(AG(w)). Likewise, &(w) = #(At(w)). By 
(3.2.2) [18] we know w € W, lies in W} if and only if At(w) c A(uné). Thus 
forwe Wi 

f(w) = #{a € A(unt) :w a € —At(€)}. 
Similarly, w € W, lies in W, , if and only if Aj (w) c Ant) = —A(un €) so 
that for w € Wi 4 one has fo(w) = #{a € —A(uNk) : w ta € wo(A*(€))}. Fix 
w € Wi. Clearly A(un €) is the disjoint union 


A(unt) = {a € Aunt) : w'a@ € wo(At (€))} 
U {a € A(unt) :wola € —wo(At(é))}. 


Define x : Af (w) — {a € A(unt): wotae ees ))} by 
part (3), ww9 € W; so that {a € A(unt) : w7la € —wo(AT(t 
Since x is bijective, we have #(A(uN €)) = #(Af(w)) + #(At(wwo)). We 
therefore have S = #A(un€) = £9(w) + (ww). This proves (4) and so completes 
the proof. 


x(a) = —a. By 
ye Mellen) 


One readily finds that Wi = {1, 0p41,Op+10p42)--- »%p+1°**On—1}- In par- 
ticular, each element w € Wi is uniquely determined by its length £(w). 
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(7.3) PROPOSITION. Let 7 be an irreducible representation of K on the com- 
plex space Z. Let yu be the highest weight of 7 with respect to At(t). Put 
D, = AMEN GN £). IfO<i< S, then HS-*(UN€, Z) @ D, is an irreducible 
representation of M with highest weight op41---op+i(ut p) — p. 


Proor. By (3.2.16) [18], H°-*(une, Z) is the multiplicity free direct sum of 
irreducible representations of M whose highest weights are of the form w(wo(,z) + 
Pk,0) — Pk,o Where £o(w) = S—%. Also, D, is a one-dimensional representation of 
M with weight —2p(une). So H5-*(anet, Z) @ D, is the multiplicity free direct 
sum of irreducible M types with highest weights of the form 


w(wo(#H) + pro) — Pr,o — 2p(une) 


where £o(w) = S — i. Since wo(px) = px.o, (7.2.2) implies these weights are of 
the form wwo(4+ px) — pe where fo(w) = S —i. By (7.2.3) and (7.2.4) we know 
wwy € Wy and £(wwo) = S — £o(w) = 7. The only element in Wj of length 7 is 
Op+1°** Opti. Consequently, 


He“ (GN, Z) @ Dy = Fru (op4i- + opsi(e + Pe) — Pr): 


Finally, we observe that every element of W} fixes the fundamental dominant 
weight wp. Since p — px is a multiple of wy, it follows that for any w € Wi we 
have w(ji+ px) ~ Pk = w(f4+p)—p. This completes the proof of the proposition. 


As discussed above, we wish to determine the multiplicity of y in 
4) (pros; (FL(A))) where (7, jz) is the CDO pair of (6.2). Since UNp =UNp_, 
(7.3) shows that the relevant E2 terms in (7.1) are direct sums of the spaces 


(7.4) Homm (Fu (op41-+- opt r(u t+ p) — p), S™(UN p_) @ Fr(A)), m2, 


where I = (0) = J+ N. Observe that if F,(A) = Fy(A) were an irreducible 
representation of M and if Hom in (7.4) were nonzero, then op41 ++: op47(u + 
p) — p— A would be a weight of S”(uM p_) for some m > 0. We now employ a 
result of Schmid [17] to show that, for certain A, S(uMp_) contains no weights 
of this form. 

Assume F(A) = F(A), ie. that the restriction of the L action on the space 
Fr (A) to the subgroup M gives an irreducible representation of M. By the 
Branching Theorem for U(p + 1) (p.186, [21]), it follows that F’,(A) = F(A) if 
and only if the coefficients a1,... ,@p in (5.7) are all zero. Thus F(A) = Fu (A) 
if and only if p’ = p+1. With p’ = p+1 the weight \ in (5.7) takes the explicit 
form 


(7.5) A= -(J+ N+ s ai wp + > ajW; 


i=pt2 i=pt2 


where J=@—land N=q-q-1. 
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(7.6) LEMMA. For m€ {1,...,q—1} we have 
(1) opp Optm (Dona bis) = OPT biwst (LP bi wp (OP bi wp 41 
+m+1 n-1 
+ te Di-1Wi + ip mgs Oi 
(2) op41 +++ Optm(P) — p = Mwy — (M+ 1)wWp41 + Wp4m4i- 


PRroor. Part (1) follows by induction on m and (2) follows immediately from 
(1). 

(7.7) PROPOSITION. Assume p’ = p+ 1 and let (r,u) be the CDO pair in 
(6.2). Then we have 


(Op41°+*Optr(u + p) — p) — A 
+, { Wy—s, — Wp + Wp41 — Wn—q’—s,41 + Wp! if £ = 1, 
Wp+1—f—s, — Wp — Wn—q’—s,41 — Wn—q! + Wn—q'4+1 + Wn—q’+e-1 ifl>1. 
PROOF. Set Y = 0p41'++Op+r. To identify the form of u, we need to consider 
the three cases given in (6.2). 
Assume N = 0. Then s, = 1 by (5.18.1), A = —Jwp41 by (7.5) and w = AZ41 
by (6.2). We first verify 


(7.8) yu) = { ce ied aie Dake 


Wp-y-1 — (J+ 1)wp —wWpy1 twpi2 if J > 0. 
If J = 0, then A = 0 and y = 1 so that y(yz) = A1 = op(p) — p by (5.2). But 
Op(p) — Pp = —Op = Wp-1 — Wp + wp41. If J > 0 then by (5.3) Ay41 = wp—y-1 — 
(J + 2)wp + wp+1 so that by (7.6.1) y(u) = wp—y-1 — (J + lwp — wp 41 + wpye- 
This proves (7.8). From (7.8) and (7.6.2) we conclude 


(7.9) (Op41-++Oper(H t+ p) — p)—A 
-{ Wp—1 — 2Wp + Wp 41 if J =0, 
Wp—J—1 — Wp — 2Wp41 + Wpp2 + Wp4J41 if J > 0. 
Since J = £—1 and s, = 1, (7.9) agrees with the expression in the proposition 
and so establishes this case. 

Assume N > 0 and s, = N +1. By (5.18.2), an_¢ = 1, and ap4; = 0 for 
2<i<N. Thus by (7.5) \= —(J +N +4 1)wp41 + Wn_q. By (6.2) and (5.3), 
B= AJEN41 = Wp-e_n — (J+ N+ 2)wp + Wn_q. By (7.6.1) we have 

w) ee if J = 0, 
YH) = ; 
Wp—e—N — (J +N+ 1)w» — Wpt1 + Wr—q’ 41 if J > 0. 
By (7.6.2) we have 
(yu + p)—p)-A 
oe if J=0, 
Wp—g—N — Wp — Wp41 — Wn—gi + Wn—q/t4i t+ Wpys4nq1 if J>O0. 


This agrees with expression given in the proposition for this case. 
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Finally, we consider the case N > 0 and s, <_N+1. As in the proof of (6.2), 
we must consider the cases s, = 1 and s, > 1 separately. Suppose first that 
S$» = 1. Then by (5.18.2) we have a,_, > 1 and by the proof of (6.2) 4 has the 


form 
n—q'—2 
w= Wp-et SY) aiyiwit (Qn—g — 2)Wn—g'—1 + Wn—g'- 
i=p 


By (7.6.1), if J = 0 then 


n—-q n—q 
¥(H) = Wp—e + ( S- ay — 2\wy = ( S- a; — 2\wp1 
i=ptl i=p+2 
n-q’-1 
as S- ajwy + (Giegr = 2)Wr—q! + Wr-q's 
i=pt+2 
and if J > 0 then 
n—q n—-q’ 
(1) = Wye + ( yo a, — 1) wp so ( S- ai — 1) wpa 
i=ptl i=p+2 
n—-q’-1 
+ ‘s ajw; + (Qn—g — 2)Wn—9 + Wn—g'4i- 
i=p+2 


By (7.6.2) and the form of \ given in (7.5) we then obtain 


Wp—1 — 2Wp + Wp41 if J = 0, 


(aut 0) —#)-r={ 
Wp— 0 — Wp — 2Wn—oi tWn—g/ti tWn—qi4e-1 if J >0. 


This proves the proposition in the case s, = 1. 
We now consider the final case 1 < s, < N+1. Then by (5.18.2) a,_9 = 


1, Qn—q’-1 = ++ = Gn—q'—s, +2 = 0 and ay,_¢/-s, 41 > 0 so that by (7.5) 
n—q'—s.41 n—q'—S.+1 
(7.10) A= -(J +N4+1+4+ S- ai wp + So AW; + Wp—q’- 
1=pt+2 i=p+2 


By the proof of (6.2) we have 
n—q'—s.—1 
= Wpt1-e-s, + > Qi41Wi + (Gpaigins41 = ee ae + Wy—g!- 
i=p 
If J = 0, then by (7.6.1) 


n—-q'—s.tl 


(Ht) =Wps.—(N+2)up+( DD ai 1upr 
t=pt2 
n—q’—s.4+1 
TE Aj;Wi — Wn—g'—s.+1 of: Wn—q’'s 


i=p+2 
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and if J > 0, then 


n—q —s.+1 
(HH) = wWpy1—-e-s. — (J + N + lwp — ( S ai p41 
i=pt+2 
n—q—S.tl 
+ S Q,Wi — Wn-q’—-s.41 a Wn—q’+1- 
i=pt2 


By (7.6.2) and the form of A given in (7.10) one has 


(Op41-+*Op+r(u + p) — p)— A 

— f Wp—s, — 2wp + Wp+1 — Wn—q'—s.t1 + Wn—q! if£=1, 
2 { Wp4+1—t—s, — Wp — Wn—q’—s.t1 — Wn—9! + Wn—g'41 + Wn—g'4e-1 if £> 1. 
This completes the proof of the proposition. 

We now describe the decomposition of S(t%M p_) into M types using a well- 
known result of Schmid [17]. This result applies to the group G; = U(p,q—1) and 
decomposes S(p; ) where py = le | :aeé ca-vxr} into K, = U(p) xU(q-1) 
types. Since M = Ky and since there exists an M isomorphism S(UN p_) = 
S(p;), we obtain an M decomposition of S(UM p_). 

Set r = min{p,q — 1} and define orthogonal roots by putting & = —(a; + 
-+++Qn—;) for 1 <i<r. Let J denote the set of integral multi-indices q= 
(j1,--- dr) with 7) > jo >--- > j, > 0. Let Emu(j) denote the irreducible 
finite-dimensional representation of M with lowest weight )7\_, J:&- 

(7.11) THEOREM [17]. The space S(UN p_) has the multiplicity free decom- 
position 

S@Np_-)= © Em(j). 
JET 

By applying the element of Wm, the Weyl group for m, of maximal length to 

the lowest weight, we find that Em (3) has highest weight 


(7.12) JrWp—r + (Jr—1 _ Sr)Wp—rti a (j1 a j2)Wp-1 oF (—ji)wpt+ 
(—J1)Wp41 aes J2)Wp+2 tees + Gri - jr )Wp+r + jrWp+r+1- 
In the case r = p (resp. r = q— 1) it is understood that the corresponding 


undefined term j,wo (resp. jg-1wWn) is deleted from (7.12). We let Fy(j) denote 
the irreducible M space with highest weight given by (7.12). 

Every irreducible representation of U(p) x U(q — 1) is equivalent to a tensor 
product of an irreducible representation of U(p) with an irreducible representa- 
tion of U(q—1) (ef. p.45, [15]). Since M = U(p) x U(q—1), we denote this fact 
by writing 


(7.13) Fu( 5 bu) =F:(Soaw) ® Fe( y bw), 


i=pt+1 
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where F(5_?_, bjw;) denotes the irreducible space for the block-diagonal sub- 
group of M isomorphic toU(p) withhighest weight }°?_, bjw; and F(t. +1 0iw) 
denotes the irreducible space for the plod diagonal subgroup a M isomorphic 
to U(q —1) with nighest weight ee p11 Oiwi- We refer to Fi()0j_, biw;) as the 
U(p) part and Py —p+1 biwi) as the U(q—1) part of the M foes in (7.13). Here 
the coefficients b; are integers for 1 < i < n—1 and are nonnegative integers for 
i #p,pt+1. Irreducible representations of U(p) are typically parametrized by the 
signature (k1,...,kp) where k; € Z,1 <j < p, satisfy kj > kp >--- > ky. The 
signature of the U(p) part in (7.13) is given by k; = }°?_, 0; and the signature 
(k1,--» ,kq—1) of the U(q — 1) part in (7.13) is given by kj = — 3-2_, bpsi- 
Define elements Z, and Z,_1 in Z(m) by Zp = S7?_, thy and Zy_1 = — 7S (q 
—i)hp4i- It follows that Z, acts on the U(p) part of (7.13) by the scalar }7?_, ib; 
and Z,-1 acts on the U(q — 1) part of (7.13) by the scalar — Ih (¢ — i)bp4i- 


(7.14) PRoposiTION. Let \ in (5.7) be a reduction point. Assume p' = p+1 
and s, >1. Let u denote the weight of the CDO pair of (6.2) and let T= J+N. 


Then for all m > 0 one has 


Homa (Fa (op41-*- Op4r(u + p) — p), S™(UN p_) @ Fr(A)) = 0. 


PROOF. Fix m > 0 and write S”(uM p_) = @;V* where V* is an irreducible 
representation space for M. As in (7.13) write V' = Vi @V} where Vj (resp. V}) 
is an irreducible representation space of U(p) (resp. U(q—1)). Since p’ = p+1, 
we have F',(\) = F\(0) @ Fo(A) so that 


S™ (iN p_) @ FL(A) = Vy @ (Vz @ F(A). 


Consequently, each irreducible component in S™(uM p_) ® F(A) must have a 
U(p) part which matches the U(p) part of an irreducible component of S™(uN 


p_). Put x = op41°+-Op4r(u + p) — p and 


(7.15) B(é,s%) = { 


By (7.7), we have 


Wp41 — Wn—q’—s. +1 + Wn—q! if 2=1, 


—Wn-q'—s.41 — Wn-q' + Wn-q'+1 + Wy—q’ 48-1 iff>1 


Fy (Wp_s, — 2wp) @ Fo(A + B(Z, sx)) if 2=1, 
710) Pu. | 
Fy (Wp41-e-8, — Wp) ® Fo(A+ BE, se)) if 2> 1. 

We first consider the case £ = 1. By (7.16) the U(p) part of Fiy(x) has 
highest weight wp_s, — 2wp if s, < p and highest weight —2w, ifs, = p. If 
8, <p then by the form of the weight in (7.12) there is no M type in S(uN p_) 
with U(p) part of the form w»_,, — 2wp. By the above discussion, Fy (x) is not 
an M type in S™(uM p_) ® F,(A) for any m > 0. If s, = p, then the only 
M type in S(uM p_) with U(p) part having highest weight —2Qw, occurs when 
p =r and j= (2,...,2). Write Fur(j = = Fi(j j) @ Fo(j j) and note that the 
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central element Z,_, acts by 2p on F,(j). Recall that the highest weights of 
the irreducible components of F(j) @ F(A) are of the form 4+ Q where Q 
is a weight of F)(j). Thus, if Fy(x) is an M type in Fy(j) @ F(A), then 
B(2,s.) must be a weight of Fo(j). But by (7.15) we find that Z,_1 acts by 
S.-Y =p—gqona weight vector with weight G(é,s,). Since 2p # p— q, we 
conclude that B(£,s,) is not a weight of F2 (5) so that Fxz(x) is not an M type 
in Fu(j) ® Fr (A). This completes the proof of the @ = 1 case. 

Assume f > 1. The highest weight of the U(p) part of Fag (x) is Wp41-¢-s, —Wp 
if@+s, < p+] and is —w, if +s, =p+1. Consider first the case +s, =p+l. 
By (7.12) the only M type in S(iMp_) with U(p) part having highest weight 
—w, occurs when p = r and 7S (1,...,1). Write Fur (j ie = Fi(j 4) @ Fah ). 
Just as in the argument for the 2 = 1 case, we know that if Fay(x) is an M 
type in Fu(j j) ® F(A), then B(é,s,.) must be a weight of F,(j). By (7.12), 
there are two possibilities for the highest weight of Fy(j). If 2p - +1l=n, then 
F(j ) has highest weight —w,,, and if 2p oa 1 <n, then Fo(j ) has highest 
weight —wp41 +Wep41. If2p+1=n, then Fo(j j) is one-dimensional with weight 
—Wp41- Since B(£,s.) #4 —wp41, we conclude that Fry(x) is not an M type 
in Fu(j) ® F(A). If 2p+1 <n, then F,(j) is equivalent to the fundamental 
representation F'(e;+---+e,) of U(q—1) with highest weight e,+---t+e,. But the 
weights of F'(e; +---+e,) are sums e;, +-+-+e;,, where 1 <i) <-+-- <i, <q -l. 
Let (’(é,s,) denote the weight for U(q — 1) corresponding to G(é,s,) in (7.15). 
Then for £ > 1 we have 


(7.17) B' (8, 84) = €q—q’—s.41 +++ + €q—q' $e—-2 + q—q' 


Since £ > 1, we have q—q’-s, +1<q-q <q-—q+£-—2 sso that (7.17) does 
not give a weight of F'(e; +--:+e,) since eg. appears twice. We conclude that 
Fu (x) is not an M type in S(UNp_) ® F(A) in the case 2p+1 <n. 

Finally, we consider the case £+ s, < p+ 1. As above, we check to see 
whether G(é,s,) is a weight of Fy(—wp41 +Wpiers,). But Fo(—wp41 + wpieis.) 
is equivalent to the fundamental representation Fe; +---+€e4s,-1) of U(q—1). 
Again by (7.17), we see that 3’(@,s,) cannot be a weight in F'(e1+---+ee4s,-1). 
We conclude that F(x) is not an M type in S(uMp_) ® Fy(A) in the case 
£+s, <p+1. This completes the proof of the proposition. 


(7.18) THEOREM. Let A in (5.7) be a reduction point. Assume p' = p+1 
and s, > 1. Set 0 = On—q'-1--*Tp—e+2- Then 


re) (prot ny (Fx (A))) = Ker(D,,,,) 


where D,,, is a nonzero CDO whose parameters are given by the CDO pairs 
(r, w) in (6.2). Consequently, TP?) (pro8sy (FL (A))) is an irreducible unitarizable 
highest weight (g, K) module with highest weight r = o(A +p) — p. 
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PRooF. Let (7, 4) be the CDO pair in (6.2) and let Z denote the irreducible 
representation space corresponding to yp. By (7.14) the E» terms in (7.1) con- 
verging to Ext? y(Z, prof; (F(A) are zero so this Ext group is zero. But the 


dimension of this Ext group is the multiplicity of Z in T°) (prov (F £())) 


gK 

alos), EK (T)) we know that 

rece) (prot ty (Fr (»))) is proper in pros «(PK (T)). Note that the assumption 
Wp), 

p’ = p+1 implies that J < q—1 if and only if £ < q’. Since A is a reduction point, 

we have £ < min{p’, g’}. Consequently, (6.5) is applicable and we conclude that 


rece) (protny(Fr())) is irreducible. 





by (6.2.15) [18]. Since Z occurs as a K type in pro 


8. A second parabolic 


In this section we state the results obtained by replacing the parabolic q(w +1) 
used in the previous sections with the parabolic q = q(w,-1). The groups L and 
M and reduction points \ are replaced with their counterparts which we now 
describe. 

Assume p > 1 and set L = S(U(p— 1) x U(q+1)). Then L is the maximal 
compact subgroup of SU(p—1,q+1) and M=LNK =U(p-1) x U(q). The 
reduction points \ for SU(p — 1,q +1) then take the form 


p-2 n—q' 
(8.1) A= SS aii - (n-pl—q' 46-14 s- ai) wp + Se aii 
i=p’ ixp—1 (=p 


where 1 < p'’< p-—1,1<q <q+land1< é< min{p’,q’}. It is understood 

that p' = p—1 iffa; =--- =a)-2 = 0 and q’ =q+1 iffa, =---=an_1 = 0. 
For A as in (8.1) we define weights \;,0 < 7 < q, by Aj = Op- ++ Op4je1 (At 

p) — p, for 1 < 7 <q, and Xo = X. Define integers J,N and s, € {0,...q— J} 

by J=q-—q+,N =p-p’—1 and 

: a if x £2, 


8.2 
oo 0 if Lx = 2, 


where © = {s > 0: Ay45 is € nonsingular}. The following result is analogous to 
(5.18). 
(8.3) PROPOSITION. Let A in (8.1) be a reduction point and suppose s, > 1. 
(1) If N =0, then s, =1. 
(2) Assume N > 0. Then s, = 1 if and only ifay > 1. If ap = 1, then 
Ss, < N41 and 
{ min(Q)+1 ifQ¥2Q, 
= 
N+1 ifQ= 9, 


where Q = {i:1<i< N-1 and ay4; > O}. 
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(8.4) PROPOSITION. Let A in (8.1) be a reduction point and assume s, > 1. 
Put 


{* if N =0, 
T= 
Op=N°**Gp—1(As + Pe) — Pk if N> 0, 
AJ+1 uf N =0, 
b= ¢ AyiNn41 if N>0 ands, =N+1, 


Op—N+s,-1°° -Op—1(As4s. + pr) — Pk if N >O ands, <N+1. 
Then (r, 4) is a CDO pair for SU(p,q) with cone data (p' + sx, q' — £, Sx). 


(8.5) THEOREM. Let A in (8.1) be a reduction point. Assume s, > 1 and 
J+N<p-l. 


(1) Define o € W by o = op_n°*+Opiy-1-. With the differential notation 
of section five we have 


Ker(d2:") = reo) (prot-xf (Fr()))). 


(2) The differential d/:\ is a multiple of the CDO D,,, whose parameters 
are given by the CDO pairs (7, ) in (8.4). 

(3) Ifq =q+1, then Te) (prota (FL(A))) is an irreducible, unitarizable 
highest weight (g,K) module with highest weight r = o(A + p) — p. 
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Dual Representations and Invariant Theory 


TUONG TON-THAT 


Dedicated to Ray A. Kunze 


ABSTRACT. Let R (resp. R’) be a completely reducible representation of 
a group G (resp. G’) on a Hilbert space H in such a way that the two 
actions commute. Then R and R’ are said to be dual representations if the 
spectral decomposition of one determines completely that of the other and 
vice-versa. The Schur-Wey] duality between symmetric and general linear 
groups, the complementary pairs of groups introduced by Moshinsky and 
Quesne, and the reductive dual pairs classified by Howe are typical examples 
of dual representations. Examples of very unusual dual representations 
which include semi-direct products of groups and infinite-dimensional Lie 
algebras are exhibited. 

Given a representation R of a group G on a Hilbert space 1 we show, 
using the theory of polynomial invariants under R, how to find its dual 
representation R’ of a group G’. Conversely, given a pair of dual represen- 
tations we use the invariants of one member to find the invariants of the 
other. 

Several theorems for dual representations and several theorems in invari- 
ant theory including some on generalized Casmimir invariants are proved. 


1. Introduction 


In the preface of his book [We] “The classical groups, their invariants and 
representations” H. Weyl wrote: “... The notion of an algebraic invariant of an 
abstract group y cannot be formulated until we have before us the concept of a 
representation of y. The problem of finding all representations of y must there- 
fore logically precede that of finding all algebraic invariants of + ...” Since it’s 
publication many treatises on invariant theory and group representations were 
written by mathematicians and physicists. One of the most striking examples of 
this intimate relation between these two concepts is illustrated by the following 
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theozem of Racah [Ra] and Chevalley [Ch]: For every semisimple Lie algebra g of 
rank n there exists a set of n algebraically independent generators of the Casimir 
invariants whose eigenvalues characterize the finite-dimensional irreducible rep- 
resentations of g. Moreover, the product of the degrees of all these generators is 
equal to the order of “the” Weyl group of g. 

The central theme of this article is in a sense to reverse Weyl’s procedure. 
This can be somewhat imprecisely stated as follows: Let R be a reducible rep- 
resentation of a Lie group G on a Hilbert space H which contains a subspace of 
polynomial functions P. Assume R is “rich enough” which, in the optimal case, 
means that R would contain all equivalence classes of irreducible representations 
of G. Find a minimal set of generators of R-invariants in P, and from this set 
derive a representation R’ on H, of a Lie group G’, which is “dual” to R. In 
general, R’ is also reducible and rich enough. Finally, use the knowledge of the 
Casimir invariants of R to find the Casimir invariants of R’. 


2. Dual Representations 


The Schur-Weyl Duality Theorem (cf. [Sc] and [We]) is the first theorem, as 
far as we know, that classifies, up to isomorphism, all irreducible representations 
of the symmetric group of degree n in terms of all irreducible rational repre- 
sentations of the general linear group of order n. This idea of characterizing 
irreducible representations of one group by that of another group was taken up 
by the physicists M. Moshinsky and C. Quesne who introduced the notion of 
complementary pairs in [MQ]. This was simultaneously generalized by K. Gross 
and R. Kunze in a series of papers starting with [GK1] and [GK2]. The idea 
was further developed in [Sa], [KV], [Ge], [EHW],.... Finally, it was R. Howe 
who gave the general definition and classification of reductive dual pairs in [Hol] 
and [Ho2] (this paper [Hol] was written in 1976 and had been widely circulated 
as a preprint). We felt, however, that this class of reductive dual pairs does not 
cover the pairs that occur in the Schur-Weyl Duality Theorem, the Peter-Weyl 
theorem, etc...., and gave a general definition of dual representation in [KT1] 
which is recalled here. 


DEFINITION 2.1. Let G and G’ be two groups. Let R (resp. R’) be a 
representation of G (resp. G’) on a vector space H such that the two actions 
commute and both representations are completely reducible. Then R and R’ are 
called dual representations if the spectral decomposition of R determines that of 
FR’ completely and vice-versa. The analogous definition of dual representations 
for Lie algebras obviously holds. 

Aside from the examples given in [KT1], it is worthwhile to mention here 
two more. The first one, given by K. Nishiyama in [Ni], establishes the cor- 
respondence between irreducible unitary representations of the orthosymplectic 
Lie super algebra osp(2/1;R) and certain finite-dimensional irreducible represen- 
tations of so(n). The second one was privately communicated to us by G. Savin 
(March 1993) and we shall describe it below: 
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Let g = g’ = s@(2,C), let C(a,y,z) be the space of rational functions in 
(x,y,z), and let H = C(z, y, z)« denote the space of K- finite vectors in C(a, y, z) 
(K = SU(2)). Let {E, F, H} denote the standard basis of g with commutation 
relations 

[H,E)=2E, ([A,F)=-2F, [E,Fl=H. 
Define the representations R and R’ on H by 
R(E) = oe R(F) = nayZ -yZ +(n-1) 


y- 
R'(E) a R'(F) = aze eo aa ae (s)” e 


= op 


1 d\n 0. 
at Gy)” Be 





where n is a positive integer. Then the R x R’ module H is decomposed into 
irreducible submodules as , 
H=)> or, 
k=1 
where Z‘*), the (k)-isotypic component, is isomorphic to Vi @ Vax, where V; 
denotes the irreducible i-dimensional module of sil(2,C). The corresponding 
highest weight vector of Z() is x*-!. 


The next class of examples, which shall be discussed from Section 4 onward, 
constitutes the central object of this article. It involves the theory of polynomial 
invariants, of which we shall discuss in the next section, the parts pertaining to 
our work. 


3. Invariant Theory 


Let V be a finite-dimensional vector over R (or C), let P(V) = S(V*) de- 
note the algebra of polynomial functions on V. Let p be a representation of a 
connected Lie group G on V. Then 


DEFINITION 3.1. A polynomial function p € P(V) is said to be p-invariant 
if p(o(g)v) = p(v) for all v € V and g € G. We have the following important 
example of polynomial invariants: 

Let g denote the Lie algebra of G, and let g* denote its dual. Take V to 
be g* and p to be the coadjoint representation of G on V. Clearly V* = g so 
that g generates P(V). It follows that for every X € g there exists a unique 
derivation of P(V) which, when restricted to g, reduces to ad(X), and that the 
invariants of the coadjoint representation of G are the elements of P(V) which are 
annihilated by these derivations. Let 4 = U(g) denote the universal enveloping 
algebra of g. Let A: S(g) — U(g) denote the symmetrizer (or canonical map), 
then is a g-module isomorphism. Let Z = Z(U) denote the center of U; i-e., 
Z={uweu:ad(X)u=0,VX € g}. Let J denote the subalgebra of all coadjoint- 
invariants in P(V) = S(g) then A(J) = Z (cf. [Di], Chap. 4 or [Go]). Elements of 
Z are called by the physicists Casimir invariants (the quadratic Casimir invariant 
is the classical Casimir operator or Casimir element). The importance of these 
Casimir invariants is due in part to the fundamental theorem of Chevalley and 
Racah stated in the introduction. 
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REMARK 3.2. If Gis a real Lie group we let g© denote the complexification 
of g and regard g as a subset of g©. Let U/© denote the universal enveloping 
algebra of g© and let U be the subalgebra (over R) generated by 1 and g. Then 
we may identify U/ with the universal enveloping algebra of g and choose a set 
of generators of the Casimir invariants in U/ so that it constitutes also a set of 
generators of the Casimir invariants in U©. This follows from the well-known fact 
that a polynomial function in several complex variables is uniquely determined 
by its restriction to the corresponding real variables. 


4. Some General Theorems on Invariant 
Theory and the Oscillator Dual Representations 


Henceforth we shall specialize to the following category of dual representa- 
tions: 

Let C"*" denote the vector space of all n x N complex matrices. Define a 
Gaussian measure js on C"* by 


(4.1) dp(Z) =2- exp[-Tr(ZZ*)]dZ , ZeEcr*’ 


where in Eq. (4.1) dZ denotes the Lebesgue measure on C”*", Let F(C”*%) 
denote the Fock space of entire functions on C"*" which are square integrable 
with respect to du. Endowed with the inner product 


(4.2) (fla) =f, S(Z)aZdul 2) f.9 € F(C™*™) 


F(C"**) has a Hilbert space structure. It can be easily verified that the inner 
product defined by Eq. (4.2) coincides with the following inner product 


(4.3) (f,9) = f(D)g9(Z) 


Z=0 

where f(D) denotes the formal power series obtained by replacing Z,; by the 
partial derivative 0/0Za;(1 << a<n,1<j< N). Note that the subspace 
P(C"*%) of all polynomial functions on C"*% is dense in F(C”*%). In the 
theorems proved in this article the notion of dual representations takes a more 
precise form. Let R (resp. R’) be an unitary representation of G (resp. G’) on 
F(C"*). As a representation space of the joint action R® R’ the Hilbert space 
F(C"*) is decomposed into an orthogonal direct sum 


(4.4) FC a Sore) 
(A) 


where in Eq. (4.4) the label (A) characterizes both an equivalence class of an 
irreducible representation Ag of G and an equivalence class of an irreducible 
representation Xi, of G’, and I) = T)(C"*%) denotes the (A)-isotypic com- 
ponent, i.e., the direct sum of all irreducible subrepresentations of R (resp. R’) 
that belong to the equivalence class Ag (resp. X%,). Moreover the restriction of 
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R®R’ to I) is irreducible; i.e, Ts) & VO) @ Wc"), where VO) (resp. 
Wc") is an irreducible Ctagdulé of class (A) (resp. Gi tnodille of class (’)). 

In [Ho3] Howe proved several general reciprocity theorems for see-saw reduc- 
tive dual pairs. However these theorems do not cover the class of dual repre- 
sentations that we will discuss later. To see this let us consider the following 
example (for more examples see [KLT]). 

Set G = U(N) and G’ = U(n) and let R’@ R denote the joint action of G’ x G 
on F(C™™”) by 

[(R’ @ R)(9', 9) f\(Z) = f(g" Za) 

for all (g’,g) € G’ x G and f € F(C"*). Then the representations R’ and R 
are dual and we have the following decomposition 


(4.5) F(C™*™*) = 5° or 
(A) 


where the label (X) denotes both a signature of an irreducible representation 
of G’ and G of the form (m:,mz,...) with m, > mg... > 0. We identify 
H := U(N —k), 1 <k < N with the subgroup of G consisting of all matrices 


ZO 
of the form | ...... .. ,y € U(N—k). Let Ry denote the restriction of R 


to AH, then it ae ered in [KLT] that the dual representation of Ry is the 
representation Ri, := R’ «x, 7, of H’ := G’ x, Hn», where H,,, is a direct 
product of k copies of a Heisenberg group acting on the Fock space F(C"**) (For 
the Fock realization of irreducible representations of Heisenberg groups see, e.g., 
[VK] Chap. 12, also this irreducible unitary representation 7, will be discussed 
again in later sections of this present article). Clearly the pair (H’, H) is not 
a reductive dual pair classified by Howe, however we are interested in having a 
reciprocity theorem for the pairs (H’, H) and (G’,G). 


THEOREM 4.1. We preserve the notations and assumptions of this section. 
Let Rg and Rg be dual representations, let H be a subgroup of G and let Ry 
denote the restriction of Rg to its subgroup H. Assume that the representation 
Ruy is also completely reducible and that the dual representation of Ry is an 
unitary representation R',, of a group H' > G' on F(C"*). Then we have the 
following decompositions of F(C"*) into isotypic components 


(4.6) F(C™%) = Ser =) ar) , 

(A) (#) 
where (A) is a common irreducible label for the pair (G’,G) and (1) is a common 
irreducible label for the pair (H’, H). 

If Xg (resp. XG.) denotes an irreducible representation of the class (A) and un 
(resp. [4,,) denotes an irreducible representation of the class (1) then the multi- 
plicity dim|Homy (jn : AlH)| of the irreducible representation iy in the restric- 
tion to H of the representation rq is equal to the multiplicity dim|Homg (AG: : 
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L'|cr)| of the irreducible representation Xi, in the restriction to G’ of the repre- 
sentation yy. 


PROOF. In Eq. (4.6) if the isotypic component Z™) is viewed, under the 
restriction of the action of G to H, as a H-submodule then obviously J?) NZ“) is 
a H-submodule of the H-module F(C”*). By the assumption of the complete 
reducibility of Ry we have the following orthogonal direct sum decomposition 


r = (HH) 
(4.7) IN ATH = Aw 
a,b 


0? 


where Vie ) is an irreducible H-module of class (2), the index “a” ranges over a 
set of cardinality equal to dim|Homy (uy : A|)] and the index b ranges over a 
set of cardinality equal to mult(Ac; Rc), the multiplicity of Ag in Rg. Similarly 
if we consider T°) 9 TZ“) as a G’-submodule of the H’-module F(C"*”) then 
we have the orthogonal direct sum decomposition 


r Qe) 
(4.8) IM nl = ews” 
a8 


where weer is an irreducible G’-module of class (X), the index a ranges over a 
set of cardinality equal to dim[Homg: (XG : p'|Gr)] and the index ( ranges over 
a set of cardinality equal to mult(y),, : Rj,,), the multiplicity of y4,, in Ri,,. By 
assumption we have 


(4.9) IM w= VAG) @ We) and T4 2 Ven @ We | 


it follows that the dimension of the Hilbert space Wc’) (see, e.g., [SK], 
Theorem 8.4) is equal to mult(Ag : Rg) and that the dimension of the Hilbert 
space V‘##) is equal to mult (j,, : Ri;,). Now by comparing the dimension of 
the Hilbert space from Eq. (4.7), Eq. (4.8), and then using Eq. (4.9) we infer 
that dim[Homy (ux : AlH)| = dim[Homag (XG : y’|G")]- Oo 


REMARK 4.2. Theorem 4.1 is very useful when dim|Homy(tn : Alr)] is 
finite; in particular when one is interested, for example as in Physics, to the 
explicit decomposition of tensor products of representations. See [Ho4], [KT2], 
and [KT3] for more discussions of this problem. 

The next theorem illustrates the intimate relation between invariant theory 
and representation theory. This idea of studying representations via invariant 
theory and vice-versa is ubiquitous in Weyl’s book [We] and was extensively 
investigated by Howe in [Hol], [Ho2], [Ho3], [Ho5], and [Ho6]; in fact, in [KLT] 
and in this present article we generalize some examples given in Howe’s articles. 

Let p be an action of a connected Lie group G on C™** then p induces a 
representation R of G on F(C™”) defined by 


[R(g)f1(Z) = f(e(g*)Z) 
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Recall that P(C"*"’), the subspace of all polynomial functions on C’™, is dense 
in F(C"*%) and a polynomial p € P(C"*%) is R (or p)-invariant if R(g)p = p 
for all g € G. Assume that the ring I of R-invariant polynomials is finitely 
generated by a set of polynomials {p;,1 < i < r}. Let I act on F(C™*%) by 
multiplication then for p € I and f € F(C"*%) we have 


[p(R(9)f)|(Z) = p(Z)f(e(g7")Z) and 

[R(g) (pf )I(Z) = (Pf )(e(9~*)Z) = plo(9-")Z) F(o(g" *)Z) 
»(Z)f(o(9~*)Z) 

for all Z ¢ C"*N. This means that the action of I on F(C"*") commutes 


with R. If q € P(C”*%) let g(D) denote the differential operator obtained by 
replacing Z,; by 0/0Z,,; then it can be shown (see, e.g., [TT 1], Eq. 2.3) that 


(4.10) R(g)q(D)R(g~') = q(D) for all gE G => R(g)q= q forall gEG, 


Eq. (4.10) means that an operator p(D) commutes with R(g) if and only if 
p is an R-invariant polynomial. It follows that together with dR’(gé(n,C)), 
the invariant polynomials and the invariant differential operators p(D), p € I, 
generate an associative algebra of operators that commute with R (dR’ is the 
differential of the representation R’). 

Let g denote the Lie algebra of G and dR denote the differential of R which 
we shall refer to as the infinitesimal action of R. Let dR, denote the Lie algebra 
of operators on F(C"*) generated by the infinitesimal action of R. Together 
with dR’(gé(n,C)) the minimal set of generators {p;,p;(D),1 < i < r} define 
a Lie algebra of operators on F(C"*) that commute with R (or equivalently 
dR). Let dRy denote this Lie algebra. Our main objective is to find the Lie 
algebra g’ and its adjoint group G’ and a representation R’ of G’ which is dual 
to R. Many examples are given in [KLT] and several more are given in later 
sections of this present article. 


Now let Up = U(Re) (resp. UR = U(RG,)) be the universal enveloping 
algebra of dR, (resp. dR,,) then the center Z(Up) (resp. Z(Up:) is called the 
algebra of all Casimir invariants of R (resp. R’). Then we have the following 
important result. 


THEOREM 4.3. Let R and R’ be two continuous unitary dual representa- 
tions of the connected Lie groups G and G’, respectively, on the complex Hilbert 
space F(C"”*%). Let W"*% denote the Weyl algebra defined by the generators 
Lois 5eo ;l<a<n,1<i< QN, tif UR and Up are mutual commutants 
in W"*N then the Casimir algebras Z(Up) and Z(Up) coincide. Moreover, if 
C € Z(Up) then in the isotypic decomposition of F(C"*N) = S37 @I™) the re- 

(A) 





striction of C to each irreducible R' ® R-submodule I) is a scalar multiple of 
the identity. 
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REMARK 4.4. From the discussion that precedes the theorem it is clear 
that Up is the commutant of Up in W"*% and in many cases, e.g., Theorems 
6.1 and 7.1, a direct verification shows that Ur is the commutant of Up in 
w"*' | If G is a reductive group then Z(Up), and hence Z(Up,), are finitely 
generated. 


PRooF. Since Up is the commutant algebra of Uz in the Weyl algebra 
wr it is clear that Z(Ug) = URNUR. By symmetry Z(UR:) = Ug NUp. 
So Z(Up) = Z(Up:). Now let C € Z(UR:) = Z(UR) and consider F(C"*) 
as a G’ x G-module. Then by hypothesis the continuous unitary representation 
R@R’ of G’ x G on F(C"*) decomposes into irreducible isotypic components 
of multiplicity one of the form F(C"*") = > @Z). Since C is a continuous 

Q) 
endomorphism of Z“) that commutes with the restriction of R’ @ R to J) 
follows from a generalized version of Schur’s lemma (cf., e.g. [Br] Theorem 4.6 
or {Na]) that the restriction of C to each irreducible R’ @ R-submodule Z™) is a 
scalar multiple of the identity. O 

In the next two sections we shall apply the procedure and theorem stated 
above to specific examples. 


5. The Harmonic Representations of U(N) 


In this section we describe the harmonic representations of U(N) or equiva- 
lently of GL(N,C). This result which we will use in Section 6 is just a slight 
generalization of several theorems in [TT2], so we will not repeat the proof (with 
its obvious modifications) and refer the interested reader to [TT2]. Let p,q, and 
n be positive integers such that p+q =n. In this section we make the additional 
assumption that N > max(p, q). 

Let E (resp. F') denote the complex vector space of all N x q (resp. p x N) 
matrices. Then GL(N,C) acts on F x E via the mapping 


((€,2),9) > (6,2)-9 = (€9,9-'x), (W(E2)EFxE). 
The bilinear form (-|-) defined by the equation 


(5.1) ((€1,21)|(€2,22)) = Tr(€iz2) + Tr(€o21), (W(é1, 21), (2,22) EF x BE), 


is clearly symmetric, nondegenerate and invariant under the action of GL(N, C) 
defined above. To every element (€,x) € Fx E corresponds a linear form (£, r)* 
defined by 


(€,2)"((6',2")) = ((E, 2) (6,2), (WE, 2’) € Fx BE). 


The correspondence (€, +) —» (€,x)* is clearly an isomorphism of the vector space 
F x E onto its dual. Let P = P(F x E) denote the algebra of all polynomial 
functions on F x FE. Then the algebras P(F) and P(E) of polynomial functions 
on F and E, respectively, can be considered as subalgebras of P. Each linear 
form (€,z)* can be considered as a homogeneous polynomial of degree 1 in P. 
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Define an action of GL(N,C) on P by 


(9° f)(é,2) = f(€g,g-*x) (V(E,x) EF x E). 


If p € P we let p(D) denote the differential operator obtained by substituting 
the partial derivatives 0/0£;; and 0/02;; for the variables €;;, x;; in p. It is well- 
known that the map p — p(D) is an isomorphism of P into the symmetric algebra 
of differential operators on P with constant coefficients. The group GL(N,C) 
acts on this algebra via (g,p(D)) — g- p(D) defined by 


(9-p(D))f =9- [P(D)(g7" - fy], (VF EP). 


A polynomial p € P (resp. a differential operator p(D)) is invariant with respect 
to the action of GL(N,C) if g-p = p (resp. g-p(D) = p(D)). One can easily verify 
that p is invariant if and only if p(D) is invariant. Let I denote the subalgebra of 
P consisting of all invariant polynomials p. Let I+ denote the subset of I which 
consists of all 7 € I such that 7(0,0) = 0. 

A polynomial function f € P is said to be harmonic if j(D)f = 0 for all 
3 €I*. It is clear that the harmonic polynomial functions form a subspace of 
P. From the invariant theory (cf. [We]) it follows that I is generated by the 
constants and by the algebraically independent polynomial functions defined by 


N 
(5.2) pig (6,2) = >> kere; , Leteps Laps ¢- 

e=1 
Let H denote the subspace (of P) of all harmonic polynomial functions, then 
obviously 


H={feP:p;(D)f=0 Wi,j,1<i<p, 1l<j<q}. 


Let I* P denote the ideal in P generated by I+ and V denote the null cone of all 
(€,2) such that f(£,r) = 0 (Vf € I*P) then obviously 


(5.3) V={(E,cr)€FxE:Ex=0}. 


Let H(V) denote the subspace of H spanned by all polynomial functions of the 
form [(€,r)*]™ for all (€,2) in V and all integers m > 0. Let Ply denote the 
algebra of all restrictions f|y, f € P then 


THEOREM 5.1. Under the assumptions and notations above the following 
statements hold: 

(i) The ideal I* P is prime. 

(ii) P=I'P@H andH=H(V),. 

(iti) The mapping j ® f > jf of 1@ P onto P is an isomorphism. 

(iv) The restriction map f — fly of H onto Ply is an isomorphism of 

GL(N, C)-modules. 


Let B, (resp. B,) denote the group of all lower triangular matrices of order 
q (resp. order p). If (r1,... , 7p) and (s1,... , Sq) are two tuples of integers such 


214 TUONG TON-THAT 


that 0 < s; < sg <...< 8, andr; >... > rp, > 0 we define the holomorphic 
characters \, : Bp > C* and A, : B,  C* by A, (bp) = bf} ...bp2, (Vbp € By) 
and A,(bg) = bj} ...093, (Vbg € B,). Set 


P(E, Aq) = {f € P(E) : f(xbq) = Aglbg) f(z); V(x, bq) € E x By} 

P(F, Ap) = {f € P(F) : f(bp€) = Aplbp) f (E); V(E, bp) € F x By} 
Let H) denote the subspace of H generated by the elements f € P(E, Ap) ® 
ae Aq) which also satisfy the condition p,;;(D)f = 9 l<i<p,1l<j<q. Let 
RO) denote the representation of GL(N,C) on H) defined by (R (g) f)(€, x) 
= f(€9,9"*x) = (9- f)(€,2). Then we have 


THEOREM 5.2. ‘The representation RO) of GL(N,C) on H is an irre- 
ducible holomorphic representation of class (A) which has signature 


(r1,.-. yp, 0,... ,0, —$1, —82;... , 8g) . 
a 
N 


Moreover, its restriction to U(N) is also irreducible with the same signature. 


6. The Oscillator Dual Representations of the Pair (U(p, q), U(N)) 


Let p,q and n be integers such that p+q =n. Let G denote the group U(N) 
and let g denote its Lie algebra. Set Gc = GL(N,C) and partition every matrix 


ZEC"*N into 
N 


— 
AP 
af Z 


Consider the Fock space ¥(C”*) and identify the vector space F x E in Section 
5 with C"*% via the mapping (€,2) > Z, with Z, = € and Z, = 2’, for all 
(€,2) € F x E. Then clearly the action of GL(N,C) on P(F x E) defined in 
Section 5 becomes 


IR @sltzy=(o-n@)=4(| 234 |), where of = oy? 


for all g € Gc and f € F(C"*%). It follows that the algebra of all Gc (or G)- 
invariant polynomials are generated by the constants and the pq algebraically 
independent polynomials 


(6.1) fipsj(Z) = (2122 );; ae Diticd: Leap TSG RG, 
s=1,...,N 


The infinitesimal action of GL(n,C) on F ) is given by 


(6.2) ae Zisng— > 1SbjSn, 
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Set 


(6.3) Dip+j = fipts (D) = me pg OF Olea SEs 
38 


and denote by P;,p4; the operator defined on F(C"*’) by multiplication by the 
polynomial f;,.;. Then an easy computation shows that 


—[Pip+i, Di p+5'] = bw Lpsj prs + 655 Liar + NO b55 


Let (g¢)° denote the Lie algebra s(n, C) with the standard basis { Hj, Fij,iz;} 
where the H;’s form a basis for a Cartan subalgebra of (gi.)°(1 < i < n—1) and if 
we set X; = Eji41 and Y; = Ei41, then (gc)° can be defined by the generators 
X;,Y;,H;,;1 <i < n-—1, which satisfy the Cartan-Weyl] relations. Define a 
representation of (g)° or su(p,q) in F(C"*%) sending the basis elements of 
(gi.)° to the operators defined by Eq. (6.1), (6.2) and (6.3) according to the 
following scheme 


(6.4) 
Hi, o Ly — Lignign, 1 <i<p-1 
Hy — Typ t+ Lptipti tN 
p+; — —Lyrj pty + Ep gtiptyti = —(Lpszpss — Lee gtiptssi)s 
7s get 
Ei; Ss Lijl<ij<piFj 


Eip+j > PipsjlSic<pl<sj<q 
Eyt ji SD Steps sg 
Eptipts > —Lptiptil Sj SGiFj 
Equation (6.4) extends to a faithful representation of (g¢)° or su(p,q) into 
F(C"*). Actually we can view this representation as dual to a tensor product 
representation as follows: 
Consider the tensor product representation of Gc x Gc (or Gx G) on F(C™™) 
defined by 
Z\ Ai91 
(Ri @ RZ )gi.g)f] 1 |---| | =f gt 
Zo Z2G3 
then the representation R© is just the restriction of R, @ R¥ to the diagonal 
subgroup Gc of Gc x Gc. This problem was studied in [KT3] for tensor products 
of the form R, ® Ro but the case R, ® RY , where RY is the contragredient 
representation to R2, can be handled in exactly the same way. Thus if we let 


q 


p 3 ; 
6.5) Ros = Y(Za)be ges Rig = — Daun gress S048 SN 
( ) B 2 Wk OZ es B »( WAZ pa Sob = 


k=1 


and let [R], [R’] denotes the matrices with (a, 9)-entry equal to Rag and Rig, 
respectively, then we know (see [KT3] Prop. 3.3) that the RC-Casimir invariants 
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are generated by the differential operators of the form 
(6.6) Tr([R]“([R']®) where d; and dz are integers >0. 


The infinitesimal of the dual action to R°, which we shall denote by L©’, where 
G’ denote the group 


Una) = {a eGL(n,c) 9 (4 seas cae 


is given by the basis {Lij3, —Lptit pj's ks —Distets 1< 4,9 < DP, 1< 
i,j’ <q, 1<k <q. The global representation L© is explicitely in [KV] on 
the Schrédinger space L?(R"*) and to obtain the representation L@’ on the 
Fock space F(C"*) one must use the unitary Bargmann-Segal transform from 
the Schrédinger space onto the Fock space; the final form of this representation 
is quite complicated and we refer the interested reader to [DES] and [HD]. Let 
X,Y,P,D denote the matrices whose entries are L;;, Lp+ip+;’, Piptk, and 
Dijp+k, Tespectively and set 





» a aan 
(6.7) [Lo] = 
-DT ; -y? 


Note that the Lj; and —Lp+i»4;’ generate over C a subalgebra Tc ~ gé(p, C) ® 
gf(q,C), the Pip+x% generate a subalgebra p, and the D,p;, generate a subal- 
gebra p_ in such a way that Jc + py + p_ gives a realization of the Lie algebra 
gé(n, C). 


THEOREM 6.1. =‘ The algebra of Casimir invariants of the representation re: 
is generated by the constants and n algebraically independent non-commutative 
differential operators Tr({[L@']*‘), 1<i<nifn<N, and1<i<N ifn>N, 
where [L°'] is given by Eq. (6.7). 


PrRoor. The Lij, —Lp+ip+j, Piptk, —Dip+re generate over R the Lie 
algebra u(p,q) whose complexification is gf(n,C). It follows from our general 
remarks in Section 3 that a set of generators of the L° Casimir invariants can 
be obtained from a set of generators of the Casimir invariants of the action of 
gé(n,C) on F(C"*%). And it is well-known that the constants and the differen- 
tial operators Tr({L°]'), 1 <i <n, generate the Casimir invariants for gé(n, C). 
0 


7. The Oscillator Dual Representations 
of the Pair (U(p,q) kr Hy, U(N — 1)) 


Set G = U(N) and let K denote the subgroup of G consisting of all matrices 
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of the form 
y ‘ 0 
bees ~yeUWW=1) - 
(rn | 
If R* denotes the restriction of R° to K, then a system of generators of the 
infinitesimal action of R = R*¢ is given by 


q 


‘ P a Z a 
7.1) R= Zi kee) RY = Z. eben 1 
(7.1) Rap >| ik CAT b >| 2) FZ ayaa a 





The dual representation of R is defined as follows. In addition to the polyno- 


mial functions f;,; defined by Eq. (6.1), which are obviously R-invariant, the 
Zin 


column vector : | is unaffected by R so that the linear polynomial function 
2nN = 

Zin, 1 <i <n, also generate R-invariant polynomials. The system {Z;y, sey 

1 <i <n, and J}, where J denotes the identity operator on F(C"*%), gener- 

ates a Heisenberg algebra of rank n. The infinitesimal dual action of R* is a 

representation of the semidirect product (sum) of u(p,q) and H, on F(C"*%) 


which is generated by the following operators 


(7.2) 





{ Lij, —Lp+i' pti» Piptk, —Dip+k as given by Eq. (6.7) 
Zin, geal sisn and I 


And we have the following commutation relations 


a) a) 


Li,j1 Zen] = je Zin [Lj a —] = Oe — 
[Lij, Zen] = O5¢Zin, (Li; Zan | Dn 





O 0 
[—Lp +i pti’s Zen] =o: pti k2pt+i!,Ns [—Lp+i' pts’) azn! = Spt k ET oy ’ 
pr) , 
[Pip+e, Zen] = 9,[—-Dip+e, Zen] = Spee z bie z : 
- “ OZin OZp+0,N 


oO , 
[P paks Bain) —bieZpik,n — 8p+keZin, |(—Dip+k, Zen | = 





The global action which is dual to R* is a representation of the semidirect 
product U(p,q) k H, on F(C"*), where H,, is a Heisenberg group of rank n. 
Set A’ = U(p,q) k H, and let L*’ denote this dual representation, then L*’ 
can be explicitly defined as in [KLT]. 


Let 
Lis = Deat,...w-1 Zisag 1 St ISP, 
(7.3) Lips! oe = Veet... .N-1 Zpti pti Sv, j" Sq, 
i.p+k = Vsa1,...,N-1 ZisZptj,51Sisplsk<ga, 
Diptk = Dsat,...w-1 DEH SES DISK SG. 
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Let X LY,P,D denote the matrices whose entries a Lis, Tgiahg ta: Pe ops and 
Dip+k, respectively, as given by Eq. (7.3) and set 


Then we have the following. 


THEOREM 7.1. The algebra of all Casimir invariant differential operators 
of the representation L*’ of the semidirect product U(p,q) * Hn, which is dual 
to the representation RX of K ~U(N —1), on F(C™™%) is generated by 

(i) the constants and the n algebraically independent Casimir operators 

Tr([L*']), 1<i<n, ifn<N. 

(ti) the constants and the N algebraically independent Casimir operators 

Tr([L*' J), 1<i<N ifN<n. 


PROOF. Clearly we have the isomorphism F(C"*”%) = F(C™*\N-)) @ 
F(C”*') so that under the restriction of R° to K the representation R* can be 
considered as the representation R@I, where R is the representation of U (N-1) 
on F(C"*(N—1)) and J is the identity representation of U(1) on F(C™*!). We 
have the orthogonal direct sum decomposition into isotypic components (cf. [KV] 
or [Ho2}) 


(7.4) F(cr*(N-)) = S_ er) (C™*(N-D) 
(u) 


where (j) denote both a signature of an irreducible U(N — 1) submodule and 
an irreducible U(p, g)-submodule ((y) is of the form (r,... ,Tp,—$1,... ,—Sq) 
where r},...,Tp and s1,...,5g are integers such that r} >... > rp > 0 and 
O<s, <...< 8, ifn = N-—1 for example). It can be shown (cf. [KLT]) that 
the representation L*’ of the semidirect product U(p,q) « Hn on F(C"*%) = 
F(C™*(N-D) @ F(C™*!) is unitary so that we also have the orthogonal direct 
sum decomposition into isotypic components 


FC") = S5 a(r) (CN) ® F(C™?)) ; 
(#) 


since H,, acts irreducibly on F(C”™*'). From [KT3] Prop. 3.3 we know that the 
Casimir invariants of R and of its dual representation coincide. But since L*’ 
and R* are dual representations and obviously the R*-Casimir invariants are 
the same as the A-Casimir invariants, Theorem 4.3 implies that the L*’-Casimir 
invariants coincide with the L-Casimir invariants. The statements i) and ii) of 
the theorem follow from Theorem 6.1. O 
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8. Conclusion 


In this article we have proven several general theorems and given several ex- 
amples of dual representations together with their Casimir invariant differential 
operators. Let us conclude by describing a general procedure for finding the dual 
representation of a representation R© of a connected Lie group G on F(C”™%). 
Suppose that G is a connected Lie group and R® is a completely reducible uni- 
tary representation of G in F(C”*”) (this situation includes the case R° is the 
restriction to the diagonal subgroup of an m-fold tensor product of representa- 
tions of G). If the identity representation of G occurs as a R-submodule and 
’ if this submodule is a space of polynomial functions then by definition these 
polynomial functions are the R@-invariants. If the algebra generated by these 
R&-invariant polynomials is finitely generated (this is the case, for example, 
when G is reductive) we may find a set of generators; in concrete cases, these 
may be obtained by computing the Clebsch-Gordan coefficients of the identity 
representations. Let {p;} denote these generators and use the set {p;,p;(D)} 
to find the dual of dR, and then, if possible, obtain the group dual action. 
To illustrate this procedure let us consider a simple example, disregarding its 
well-known results which will be derived subsequently. 

Let R denote the representation of GL(N,C) (or U(N)), N > 2, on the Fock 
space F(C!*") defined by 


Ry) f(Z) = f(Zy), (Wf € F(C**™), y € GL(N,C)) . 
Then F(C!*") is decomposed into an orthogonal direct sum decomposition 


F(C!*%) = s @P™(C*%) , 


m=0,1,2,... 


where P™(C!*%) denote the vector space of all homogeneous polynomials of 
degree m in N variables Z,,... , Zy. The dual representation to R is the repre- 
sentation L of GL(1,C) which is obtained by left translation on F(C!*%). Each 
P™(C!*%) is therefore an isotypic component. The submodule P°(C!*%) = C 
is the identity component of R. Thus the only R-invariants are the scalars which 
is the uninteresting case. Now let G = SO(N) and denote the restriction of R 
to G by R°. Each R°-submodule P™(C!*) can be further decomposed into 
irreducible R@-submodules as 


(8.1) Pre’) 2 ys ep H™-2 , 
i=0,... ,[m/2] 


where in Eq. (8.1) p(Z) = aS Z? and H™~* is the space of all harmonic 
polynomials of degree m — 21, 1 < i < [m/2]. Thus we have the spectral de- 
composition of the R°-submodule P™(C!*™) given in terms of signatures of 
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irreducible representations of G as 


(8.2) (m,0,... ,0) @(m— 2,0,... ,0)@--- @ (m — 2[m/2j,0,... ,0) . 
Nee preemie ce eA, 
[N/2] [N/2] [N/2] 


In Eq. (8.2) we see that the identity representation (signature (0,0,... ,0)) 
occurs when m is even. Therefore the R°-submodule which corresponds to the 
identity subrepresentation is the algebra of all polynomial functions generated 
by {p?, m=0,2,4---}. This is precisely the algebra of R@-invariants which is 


generated by p. Thus if we set P = —p, D = p(D) and H =4 >> Ze +2N 
j=l ‘ 


then {P, D, H} generates the infinitesimal dual action to R°. Clearly {P, D, H} 
forms a basis of s@(2,R) (or sp(2,R)), and this infinitesimal action eventually 
leads to the group dual action (see, e.g., {KLT] for details). 

Finally let us remark that in this article we concentrated on the pair (U(p,q), 
U(N)) since this dual action arises naturally from the harmonic representations 
of U(N) studied in Section 5, but if we restrict those representations of U(N) to 
a subgroup such as O(N) or Sp(N) (N even) then we will obtain many equally 
interesting dual representations. 
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ABSTRACT. Harish-Chandra defined an algebra homomorphism from the 
polynomial differential operators on a complex reductive Lie algebra that 
are invariant under the adjoint group into the polynomial differential oper- 
ators on a Cartan subalgebra that are invariant under the Weyl group. We 
give a formula for this homomorphism analogous to the Harish-Chandra 
homomorphism from the center of the universal enveloping algebra onto 
the Weyl group invariants in the symmetric algebra of a Cartan subalge- 
bra. Our construction involves the analysis of a certain D-module on the 
Lie algebra which is of independent interest. 


Introduction 


If V is a finite-dimensional vector space over C then we denote by D(V) the 
algebra of all differential operators on V with polynomial coefficients. If G is 
a subgroup of GL(V) then we denote by D(V)° the subalgebra of all elements 
of D(V) that commute with the action of G on functions. In this article we 
consider the case when V = g is a reductive Lie algebra over C. Let G denote 
the identity component of the group of automorphisms of g that act by the 
identity on the center of g. Let h be a Cartan subalgebra of g and let W denote 
the Weyl group of g realized as {g),/9 € G,gh =}. In [HC2], Harish-Chandra 
proved that there exists an algebra homomorphism 6 : D(g)° —+ D(h)” with 
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the following properties. If f is a G-invariant polynomial on g looked upon as 
a differential operator under multiplication then 6(f) = f\y. If D is a constant 
coefficient G-invariant differential operator on g looked upon as an element of 
the symmetric algebra of g (in turn identified with the polynomials on g*) then 
6(D) is the restriction to h* relative to the decomposition g = ) @ (, g]. The 
kernel of 6 is the space of D € D(g)© such that Df = 0 for any G-invariant 
polynomial f on g. Harish-Chandra’s proof of the existence of 6 uses analysis to 
show that certain operators that a priori have rational coefficients actually have 
polynomial coefficients. 

The main purpose of this paper is to give a completely algebraic construction 
of 6 analogous to that of the Harish-Chandra isomorphism from the center of 
the enveloping algebra to the Wey! group invariants in the enveloping algebra 
of h. Our proof that our construction does indeed yield 6 has two ingredients. 
The first is the recent theorem (see [LS] and [Wal]) that D(h)™ is generated 
by the W-invariant polynomials and the W-invariant constant coefficient differ- 
ential operators on h. The second is the analysis of a certain module, M, for 
D(g) analogous to a Verma module for g. To give an idea of the delicacy of our 
result we show how it implies Freudenthal’s “strange formula” for the square 
of the norm of 9 relative to the Killing form on a semisimple Lie algebra. We 
actually give two proofs of the main result. The second involves applying the 
Zuckerman functors to M thereby constructing a (D(g),G)-module. Under suit- 
able hypotheses we prove that the module is holonomic. In Section 8 we give a 
conjectural decomposition of this module of the form 


GD (dim )N” 


yew 


where the N” are pairwise non-isomorphic irreducible D(g)-modules and W is 
the set of equivalence classes of irreducible finite-dimensional representations of 
W. We check this conjecture for g = slg. 


It is our pleasure to dedicate this article to Ray Kunze in commemoration of 
his 65th birthday. 


1. A homomorphism due to Harish-Chandra 


Let g be a reductive Lie algebra over C with adjoint group G. The adjoint 
action of G on g induces actions on the algebra of polynomial functions O(g) 
and hence on the (non-commutative) algebra of polynomial differential operators 
D(g) ¢ Endc(O(g)). Let § be a Cartan subalgebra of g, H the corresponding 
Cartan subgroup of G. Then the Weyl group W = Ng(H)/H acts on §, hence on 
O(h) and on D(h). In [HC2], Harish-Chandra constructed an algebra homomor- 
phism 6 : D(g)° —> D(h)” which extends the Chevalley restriction mappings 
O(g)% — O(b)* and S(g)° — S(b)™. 

We recall briefly the construction of 6. Let ® be the root system of (g,4). 
Choose ®,, a system of positive roots. Set 7 = Tea, a and h/ = {he 
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h | w(h) # O}. Then bh’ is a W-invariant affine open subset of h on which 
W acts freely. Therefore, if D € D(g)° then there exists a unique element 
6'(D) € D(b’)™ such that 


5'(D)(fip) = D(f)ip for f € O(g)°. 


Now define 6(D) = 76’(D)x~1. A priori, 6 is a homomorphism of D(g)° into 
D(h’)”. Harish-Chandra’s formula in [HC1] for the radial component of a reduc- 
tive Lie algebra implies that 6 extends the restriction mapping S(g)° — S(b)”. 
In [HC2], Harish-Chandra proved that for each D € P(g), 6(D) is regular on b, 
ie. 6(D) € D(h)”. 

The kernel of 6 is the ideal J = {D € D(g)° | D(f) = 0 for all f € O(g)°}. 
Recently, Levasseur and Stafford [LS] proved that the following sequence is ex- 
act: 

0 I — D(g)° D(H)” + 0. 
More precisely they showed that if A is the subalgebra of D(g)° generated by 
the subspaces O(g)° and S(g)° then 6(A) = D(h)™. This is a consequence of 
the following result (cf. [LS, Theorem 3}): 


THEOREM 1.1. Let V be a finite-dimensional vector space over C and let W 
be a finite group acting linearly on V. Then D(V)™ is generated (as an algebra) 
by the subspaces O(V)“ and S(V)”. 


We will use this theorem and the construction of the next section to give an 
algebraic proof of the fact that 6(D(g)°) c D(h)”. 
2. An algebraic construction of 6 


We retain the notation of the previous section. Again, we fix ®;,, a system 
of positive roots for ®. Set ny = d’,c@, Ba and n_ = Dade g—q- Then we 
have a triangular decomposition g = n_@h @n,. This decomposition gives also 
a decomposition of the dual g* =n * @h* G@n,t. We note that if V is a finite- 
dimensional vector space then D(V) = S(V*)S(V) and also D(V) = S(V)S(V*). 
Thus, we may write D(g) in the following way: 


D(g) = S(g*)S(g) 
= $(n_*)S(n_) $(*)S(b) S(n*)S(n4) 
= S(n_)S(n*) $(*)S(b) S(n*)S(n,). 


This provides a linear isomorphism 
D(g) = S(n,* @n_) ® D(b) ® S(n* ny) 
which induces a decomposition 


D(g) = D(b) @ (ng G n_)D(g) + D(g)(n7 Gn4)). 
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Let p denote the corresponding projection of D(g) onto D(h). Clearly, the map 
p extends the restriction mappings on O(g) and on S(g). Our goal is to prove 
the following 


THEOREM 2.1. The restriction of p to D(g)° is equal to 6. 
REMARK. This proves in particular that 6(D(g)°) ¢ D(h)”. 
The proof depends on three lemmas. 


LEMMA 2.2. The restriction of p to D(g)" is an algebra homomorphism into 
D(b). 


PROOF. Let a,...,@, be the positive roots. Choose a basis X,...,X;, of 
n, with X; € ga, and a basis Yj,...,Y, of n_ with Y; € g_o,. Let 1,...,2, 
and yi,...,Yy, denote the corresponding linear coordinates on ny and n_. If 
I = (iy,... ,ip) € N” we define z! = x--- xr, X! = X}..- X* and similarly 
y! and Y!. 

The elements 27Y¥7 y/X" (I, J,I', J’ € N’), form a basis of D(g) as a (left 
or right) D(h)-module. The submodule D(g)” is spanned by the elements of this 
basis with 3°, (ix + j, Jak = 3°, (ie + t,)a%. This clearly implies 


D(a)" n((nt @n_)D(g) + D(g)(nt @n4)) = D(a)" ND(g)(n.* @ m4) 
= D(g)" 1 (nyt @ n_)D(g). 
Now, if Dj, D2 € D(g)” then 
D,Dz = Dp(D2) (mod D(g)(n* 6 n+)) 
and finally, 
D, D2 = p(Di)p(D2) (mod (n, @ n_)D(g) + D(g)(n* @n+)). 
This is the content of the lemma. 0 
LEMMA 2.3. p(D(g)°) is contained in D(h)™. 


ProoF. If a € ® we denote by sg the reflection in W associated to a. Let 
& denote the system of simple roots in 64. The Weyl group W is generated by 
the sq for a € L. Therefore, the Lemma follows if we show that p(D(g)°) is 
contained in the sg-invariants in D(h) for a € &. 

Fix an element a € DY. Letm=gg@Ph@ge. Set up = do peo4\{a} 9, 
and u_ = yo Eo4\{a} g-g. Then g =u_ @m@ uy and we obtain a direct sum 
decomposition 


D(g) = D(m) & (uy @u_)D(g) + D(g)(u* Guy)). 


Let q be the corresponding projection of D(g) onto D(m). Let M be the adjoint 
group of the reductive Lie algebra m. We have [m,u,] C uy and [m,u_] C u_. 
This implies that q(D(g)*) is contained in D(m)™”. Since po gq = p, we may 
assume that g =m, ie. 6, = {a}. 
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We are reduced to the case when g = [g,g] = sla. Let X1,X2,X3 be an 
orthonormal basis of g with respect to the Killing form. Let 1,,272,23 denote 
the corresponding linear coordinates. Using classical invariant theory we see 
that D(g)° is generated as an algebra by the Laplacian A = X? + X? + X?, the 
quadratic form r? = x?+23+23 and the Euler operator F = 2X1 +22X2+23X3. 
Set e = A, f = —jr? and h = —-E-— 3. Then h,e,f satisfy the relations 
le, f] = h, [h,e] = 2e, [h, f] = —2f. Thus, h,e, f span a three dimensional 
simple Lie algebra s and we obtain an isomorphism of algebras D(g)° = U(s). 
(Here U(s) denotes the universal enveloping algebra of s.) In particular, D(g)? 
is generated as an algebra by O(g)© = C[r?] and S(g)° = C[A] in this case. 
Hence p(D(g)°) is contained in D(h)”. O 

So far we have proved that p is an algebra homomorphism of D(g)° into 
D(b)” which is equal to 6 on the subalgebra A of D(g)° generated by O(g)° 
and S(g)°. Let 7 be the kernel of the homomorphism p : D(g)° —> D(h)”. In 
light of Theorem 1.1 we have D(g)° = 7 +A. Therefore, if we show that I = J 
then Theorem 2.1 follows. 

Recall that the differential of the action of G on g provides a Lie algebra 
homomorphism 7 : g + Der(O(g)) C P(g). Clearly, P(g)7(g) M P(g)? C T. 


LEMMA 2.4. D(g)t(g) N D(g)? CJ. 


Assuming the assertion of Lemma 2.4 we now complete the proof of Theorem 
2.1. 


PROOF OF THEOREM 2.1. Let d be the discriminant of g, i.e. d is the non- 
zero polynomial on g such that 


det(t — ad(X)) = d(X)t' + terms of higher order in t 


for all X € g (with ¢ an indeterminate). Then d € O(g)° and dj, = 7”. In [Wal] 
we proved that (D(g)r(g) N D(g)°) {a = Zia. Here the subscript [d] means lo- 
calization at d. Lemma 2.4 implies Ziq) C Jaq. Now we prove that the inclusion 
is in fact an equality. We note that D(h’)” = D(h)/%> and that the homo- 
morphism 6’ : D(g)° —+ D(h)(¥2) has a natural extension to a homomorphism 
6’: D(9) fy — D(b) fra). Theorem 1.1 together with Harish-Chandra’s formula 
for 6’ on A implies that the following sequence is exact: 


re 
C= Lig D(a) — D(b) he, — 0. 


We observe that Db) iva is a simple algebra. This follows easily from the fact 
that the algebra D(h)™ is simple (cf. [Wal, Lemma 1.2]). Thus, Tia is a maximal 
ideal of D(g)f- Hence T;4, = Jaq. This implies T= 7. O 

We will give two proofs of Lemma 2.4 in sections 4 and 5. In the next section 
we want to show that Lemma 2.4 surprisingly implies Freudenthal’s “strange 
formula.” 
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3. Freudenthal’s “strange formula” 


In this section we assume that g is semisimple. J.et B denote the Killing form 
on g. If A € h* then define Hy € h by B(A)y,h) » A(h) for hh Eh. Let (, ) be 
the bilinear form on §* given by (A, #) = B( A), i/,,) for 4, u € b*. Freudenthal 
proved the following formula, which is now known as Freudenthal’s “strange 
formula”: 


THEOREM 3.1. Let p= 4 Dace, o Then 


1 
dim g = 57 (¢,). 


We want to show that this formula is implied by Lemma 2.4. 


LEMMA 3.2. Let 2 € U(g) be the Casimir element in the universal enveloping 
algebra of g. Then 


1 
p(7(2)) = 6(p, p -5 Xe a) — 5. 
2 cen 
PRooF. Let X},...,X, be an orthonormal basis of g (with respect to B). 


Then 2 = $5, X2. Hence 


7(Q) =P (Xx) 


si [Xe Xi XF [Xk XG] 


a,g,k 

= $0 X}B([Xe, Xi], Xj)[Xe, Xj] + D5 XP XP Xe, Xi] [Xe, XG] 
4,9,k 4,9,k 

= SO XP + D0 XT XTX, Xil[Xe X)] 
a 4,9,K 

=E+D 


with © the Euler operator on g. In order to calculate p(D) we choose another 
basis of g: For a € ®4 let Xq € ga and X_g € g_q such that B(Xq, X_q) = 
1. Furthermore let H,,...,H,; be an orthonormal basis of h. Then, modulo 
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(n,* @ n_)D(g) + D(g)(n* @ n,), we obtain 


D= S> XEX§[Xe, Xal[Xe, Xp] +2 D> HTXG[Xe, Hil [Xe, Xo] 


a,B,k B,i,k 
+ 5° HTH} (Xe, H. ‘[Xe, 5] 
i,7,k 
= SD XLNGXe XallXerXol+2 DD ALG, Ml%e, Xol 
a, 8-<0,k B~0,i,k 
= ye B [Xx Xa], Xs) B([Xx, Xe], X-g) 
a,B~<0,k 
+ > B (Xn, X X_o)B([Xx, Xa], X-g) 
a,B~<0,k 
—2 $0 HBA) B(X, X-p)[Xx, Xo] 
B~<0,i,k 
= SO BIXp,(X-, Xall,X-a) + S> B(Ha, Ha) +2>_ Be 
a,B~<0 a,B~<0O B~<O 
=C,+Cot+ E 


with C1, C2 constants and E’ the Euler operator on 4. Clearly, C2 = 4(p, p). It 
remains to determine C,. We note that in U(g) we have the identity 


> XeX- p= 59- H, 3 


B~<0 


Hence i ; 
YX, [X-g, Xa]] = gna — (0,0) Xa — 9 (0 0) Xa. 
80 
This now implies 
1 1 
C1 = 3 [P+ + 2(p, p) — 3 2 (a.0). 
Finally, we obtain 


p(7(2)) = p(E) + p(D) 
oS et eee ee 3 
1 1 
= (0,0) — 5 Dd. (aa) - 5|®+1- 
ach, 
This completes the proof. O 
On the other hand, Lemma 2.4 implies p(7({)) = 0. Thus, we have proved 
the formula 
p) = Do (a,a) + |4. 
aed 
Freudenthal’s formula is now a consequence of the following nice identity (cf. 


[Br]): 
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LEMMA 3.3. 
dim h = S>(a,a). 
acd 
ProorF. The dimension of h is the rank of the Gram matrix A = [(a, B)Ja,gee. 
We now observe that 


(a, 8) = tr ad(H,) ad(Ha) = 9° y(Ha)y(Hs) = do (a, 7) (7, 8)- 
ved ye@ 
Thus, the matrix A (looked upon as a linear operator on C!®!) is idempotent. 
Hence the rank of A is equal to the trace of A. This implies the asserted iden- 
tity. O 


4. A (g,D(g)°)-bimodule and the proof of the main lemma 


The key to the proof of Lemma 2.4 is a certain D(g)-module, which we now 
introduce. Let S(h @n_* @n,) act on C by evaluation at 0. (Here we identify 
S(h@n* @n,) with the polynomials on h* x n_ x n,*.) Then we define M to 
be the D(g)-(left) module 


M= D(g) @s(hon_*@n,) C. 
We also look upon M as a g-module via 7. Note that this g-action commutes 
with the action of the invariant differential operators D(g)°. 

The weight structure of the g-module M will be essential. 

LEMMA 4.1. Leta ,,...,a, be the (distinct) elements of ®,. Let p = 5 yo a. 
Choose a basis X1,...,X, of nz with Xi € ga, and a basis Yi,...,Y, of n_ 
with Y; € g-a,- Let x1,...,2, and y1,...,y, denote the corresponding linear 
coordinates onn, andn_. Then 

i) The g-module M is h-diagonalizable, i.e. is M admits a decompostion 
M= Brey: My into weight spaces My. 
ii) If \ € h* then the weight space My is given as 


My = B O(h) a ¥/ @1. 
I,JEN" 
—2p-Yo, (te tik \aR=A 


PROOF. The elements z7Y¥ @1 (I,J €N’*), form a basis of M as a O(h)- 
module. For h € h we calculate the action of 7(h) on an element of this basis. 
First. we consider the special case when I = J = 0. We note that r(h) = 
— ip Ok (h) (EX — YeYe). Thus, 


T(h) 1@1= Sax (h) yn Ve Ql 
k 


= S> an (h) (Yeve = 1) Ql 
k 


= —2p(h) @1. 
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In the general case we write 
r(h) 2! YY @1=2'Y"r(h) @14 [r(h), 21 YJ @1 
A straightforward verification shows that 


[r(h), 2'Y7] @1=—S (ik + jedog(h)z'Y! @ 1. 
k 


Hence we see that z7Y/ @ 1 is a weight vector of weight -2p — oy (te + Je) oe 
The claims of the lemma now follow. OU 

The homomorphism p : D(g)° —+ D(h)™ enters the picture when we look at 
the action of D(g)° on the —2p-weight space M_2, = O(h) @1 

LEMMA 4.2 If D € D(g)° and f € O(h) then Df @1 = (p(D)(f))@1. 


Proor. If D € D(g)° then D = p(D) + D’ with D’ € D(g)(n_ @n,). Hence 
Df ®1=p(D) f @1. We note that D(h) 1@1 = D(h) @s(4) C = O(h) with the 
usual action. This means that p(D) f ®@1=(p(D)(f))@1. O 

The two preceeding lemmas provide all the information on M that we need 
to prove Lemma 2.4. 


PROOF OF LEMMA 2.4. Let B be a non-degenerate invariant bilinear form 
on g. We assume that B restricts to the Killing form on [g,g]. Let X1,...,Xn 
be an orthonormal basis of g with respect to B. 

Let D € D(g)r(g) ND(g)%. Write D = >, Dir(X;). Let w be the linear map 
from g to D(g) defined by sending X; to Dj. 


CLAIM. We may assume that w € Homg(g, P(g)), i-e. g¥(X)g7! = 
w(Ad(g)X) forg € G,X €g. 


PROOF OF THE CLAIM. If g € G let Ad(g)X; = )0, a;i(g)Xj. Since D € 
D(g)° we have D = 5°, , a;:(9)gDig~*t(Xi). Let K be a maximal compact 
subgroup of G and let dk We a left invariant normalized measure on K. Define 


Di= J ak Yay Hed, }. 


(Those that prefer algebra can use an appropriate Reynolds operator in place 
of the integration.) Then D = 0, Dit(X;,) and gDig~! = Yo, aij(g7!)D; = 
dX; 4ji(9)D;j. The claim follows. O 

Choose for each a € ® an element Xq € ga such that B(X,, Xa) = 1. Let 
Ay,..., H; be an orthonormal basis of h with respect to B. Then 


D= ZF Xq,X;)Dit(Xa) + DBs X;)Dit(H;) 


Suk a ra) + Ew H;). 
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For f € O(h) we calculate 


Df@1= > ¥(Xa)t(X-a)f ®1- 25° p(H;)Y(H,)f @1. 
a>O j 

By the equivariance of = we have ~(Xq.)f @ 1 € M_2,4.. Lemma 4.1 implies 

M_2p)4a = 0 for a > 0. Hence 


W(Xoq)T(X_a)f @l= (W(Xo),T(X_-a)|f Q@l= v( AA) f @ 1. 


Thus, we obtain 


Df®1= S> o(A)v(Hi)f @1- 27 o(Hs)¥(Hs)F @1 


t,a>0 


= 0. 


Lemma 4.2 now implies p(D) =0. O 


5. Applying Zuckerman functors to M 


In the sequel, let r = 3 dim(g/ h). In this section we give a second proof of 
Lemma 2.4 which involves applying the Zuckerman functor (TG)" =I" to the 
module M. We start with the following observation. 


LEMMA 5.1. U(g)O(h) @1 is a (g, D(g)°)-bimodule summand of M isomor- 
phic to M(—2p)@O(h) with g acting on the Verma module M(—2p) with highest 
weight —2p and D(g)° acting on O(h) via p. 


Proor. If f € O(h), f #0, then 7(U(g))f @1 is a highest weight module 
with highest weight —29. We note that every highest weight module with highest 
weight —2p is irreducible. This implies that 7(U(g))O(h) @1 = M(—2p) @O(b). 
(The action of D(g)° is the predicted one since the D(g)°-action commutes with 
the g-action.) Next we observe that Hom,(M(—2p), M)®M(-—2p) is asummand 
of M. Since O(h) @ 1 is the full —2p-weight space we obtain the result. O 


OUTLINE OF A PROOF OF LEMMA 2.4. Let s be the (unique) element of 
the Weyl group W such that s(@,) = —6,. Then s(—29+ p)+p = 0. This 
implies that if we apply the Zuckerman functor (TY)" = I" to the irreducible 
Verma module M(—2p) then we obtain the trivial representation C (cf. [RRG, 
6.6]). The idea is now to apply I” to the module M (looked upon as a (g, H)- 
module). We will show that ['’(M) admits a natural D(g)-module structure such 
that the g-action is the one induced by rt. Then I'7(M) contains a (g, P(g)°)- 
bimodule summand isomorphic to C ® O(h) with g acting trivially on C and 
D(g)° acting on O(h) via p. The assertion of Lemma 2.4 is now reduced to 
a triviality. If D € D(g)? M D(g)r(g) then we write D = $7; Dir(X;) with 
D; € D(g),X; € g. Applying D to1@f € C®O(h) c T"(M) we obtain 
18 ((D)(f)) =H, D(X1e f)=0. O 

We now give the details needed in the proof above. We start by recalling the 
construction of the Zuckerman functors (cf. [Wa2]). For the moment let G > H 
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be arbitrary reductive linear algebraic groups over C with Lie algebras g D h. 
Assume that G is given as a closed subgroup of a linear algebraic group G. Let g 
denote the Lie algebra of G. A module M for g and H is called a (g, H)-module 
if the following three conditions are satisfied: 


i) If X €g,h € A then hXm = Ad(h)Xhm for all m € M; 
ii) As an H-module, M is locally finite, ie. M is the union of finite- 
dimensional algebraic H-modules; 
iii) If V Cc M is a finite-dimensional H-submodule then V is h-invariant 
and the action of h on V is the action induced by the differential of the 
action of H. 


Morphisms of (g, H)-modules are linear maps that are both g- and H-module 
homomorphisms. 

Let M be a (g, H)-module. Let O(G) denote the ring of regular functions on 
G. If X € g then we define an endomorphism p(X) of M @ O(G) as follows. 
First we identify M @ O(G) with the space of mappings y : G —> M such that 
p(G) is contained in a finite-dimensional subspace V of M and y:G —> Visa 
morphism of algebraic varieties. (If m € M, f € O(G) then we set (m@ f)(g) = 
f(g)m for g € G.) Then we define 


MX)p(g) =Ad(g)X~(g) forpe M@O(G),gEG. 


Clearly, if X,Y € g then u([X, Y]) = [u(X), u(Y)]. 

Let 7 denote the (g, H)-action on M. On O(G) we have two natural (g, G)- 
module structures: Set p:(g) f(x) = f(g~!x), pr(g)f(xg) for f € O(G),9,xr EG. 
We look upon M ® O(G) as a (g, H)-module under 7 ® p;. One can check that 
for X € g, the endomorphism p(X) of M @ O(G) is a (g, H)-homomorphism. 
Furthermore, for each g € G, J ® p,(g) is a (g, H)-homomorphism of M @ O(G) 
to itself. 

We now consider the relative Lie algebra cohomology of (g, H) with coefficients 
in M @ O(G). Recall that the complex is given as 


Ci(g, H; M @O(G)) = Homy(\7(g/h), M @ O(G)) 


with the usual coboundary operator of relative Lie algebra cohomology (cf. 
[BW]). We define 

T9(M) = H?(g,H;M @O(G)). 
For each X € g the endomorphism p(X) induces an endomorphism ji(X) on 
I¥(M). This defines a g-module structure on I9(M). The G-action IJ @ p, on 
M ® O(g) induces a G-module structure on T7(M). With respect to these two 
actions T7(M) is a (g, G)-module (cf. [RRG, 6.3]). 

Next we introduce a certain category of D-modules. Let V be a finite- 
dimensional vector space over C. Let H be aclosed reductive subgroup of GL(V). 
A module M for D(V) and H is called a (D(V), H)-module if the following two 
conditions hold: 


i) If DE D(V), hE H then hDm = (hDh—!)hm for allm € M; 
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ii) As an H-module, M is locally finite. 


Let § be the Lie algebra of H. A (D(V), H)-module M is called compatible, if 
the induced h-action is given via rT : h ~ D(V) and if M is a (h, H)-module 
under this action. 


LEMMA 5.2. Let g Dh be reductive Lie algebras over C with adjoint groups 
GDH. Let M be a compatible (P(g), H)-module. Then T(M) admits a natural 
structure of a compatible (D(g), G)-module. 


PROOF. We consider the Heisenberg algebra a = g @ g* @ C1 C D(g). Note 
that a is normalized by the action of G. Thus, me may look upon M as a 
(g k a, H)-module. (G C G x A with A the Heisenberg group corresponding to 
a.) We obtain a g x a-action on I7(M). In particular T3(M) is a U(a)-module. 
Let c denote the central element 1 € a. There is a canonical isomorphism of 
algebras D(g) = U(a)/U(a)(c — 1). The U(a)-action on T3(M) induces a D(g)- 
action if z(c) = 1 on M @ O(G). But this is clear since Ad(g)c = c for g EG 
and ¢ acts trivially on M. 

We now show that the g-action on I? (M/) is equal to the action of r(g) C D(g). 
Let X1,...,Xn be a basis of g. If X € g then r(X) = — 35, X7[X, Xi] € D(g). 
We calculate the action of —$°, X7[X, X;] € U(a) on M @ O(G): 


uu(— >: XTX, Xi])o(9) = - a Ad(g)X} Ad(g)[X, XiJy(g) 


—_ ye X}{Ad(g)X, XiJe(g) 


a 


= t(Ad(9)X) (9) 


for p € M ® O(G),g € G. Hence the result. O 

We return to our previous situation. That is, h is a Cartan subalgebra of g 
and M is the module defined in Section 4. Then M is a compatible (D(g), H)- 
module and we obtain (P(g), G)-modules T?(M) with the properties needed in 
the outline of the proof of Lemma 2.4 above. In the next sections we continue 
the study of these modules. 


6. The case of g = sil 


If g = slo then it is possible to describe M very explicitly. The key is the 
analysis of the formal character of M, which we will introduce for general g. 

We decompose the (g, H)-module M into modules from the category O. Let 
E € D(g) be the Euler operator. Then EF acts semisimply with integer eigenval- 
ues on M. If i € Z then we define 


M*={meM | Em=(i-r)m}. 


Since FE commutes with the adjoint vector fields r(g) we have a decomposition 
AM = @, M? into g-submodules. These modules are from the category O. In 


INVARIANT DIFFERENTIAL OPERATORS 235 


particular, their weight spaces are finite-dimensional. Recall that the formal 
character of M? is given as 


ch M‘ = 5° (dim M})e* 
AEh* 
We also consider a graded formal character for the graded object M. Set 
ch(M, q) =— s (dim M3) 
i XAEh* 


with q an indeterminate. Using Lemma 4.1 we obtain a presentation of ch(M,q) 
as a rational function 


e7P 
(lo) Tec ger) ag e-2), 





ch(M, q) = 


Let us now compute this character for g = sly. Set r = e~%. The key is to 
expand 


l-« ky —-k _ gq kt} —k+2 k 
~~ = Si rk(q +97 ht? —... 4g) 
Gaya) ~ 2 

k>0 
ee 
k>0 


=Soatg k ye a ania 





k>0 1+q 
Thus, 
z gktl gn* gett 
Ge ee ae 
MM Ge aaa eae) fies Gee? isa 


From this formula we can read the explicit structure of M as a (g, P(g)°)- 
bimodule. Recall that D(g)° = U(s) where s = Ch ®Ce @ Cf is the three 
dimensional simple Lie algebra introduced in the proof of Lemma 2.3. The spaces 
M? are the weight spaces of M as a s-module with respect to the semisimple 


element h = —E — 3. The eigenvalue of h on M? is —i — 5: We now see that 
1 
M = @ M(-29 - ka) ® (M(-k - 5) ® Mk =\) 
k>0 2 


a (g, P(g)%)-bimodule. Here oe . — 3) (resp. M(k — 3)) is the Verma 
ee for s with highest weight —k — 3 (resp. k — $). For the module I'?(M) 


we obtain 
3 1 
T*(M) = @ Fx @ (M(-k - 5) ® M(k- =)) 
Sp 2 2 
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where Fy, denotes the irreducible g-module of dimension 2k + 1. It is interesting 
to compare this decomposition with the decomposition of the D(g)-module O(g) 
(with the usual action). The graded character for O(g) = @; O*(g) is given as 


1 
(1 — q)(1 — gx)(1 — ga-*) 


=Si(e kta +... 42%), x 5: 
k>0 1—q 





ch(O(g),q) = 





Hence 
3 


O(g) = @D Fx @ M(-k — 3) 


k>0 
as a (g, D(g)°)-bimodule. The D(g)-module I'(M) seems twice as big as the 
module O(g). In Section 8 we will prove the following 


THEOREM 6.1. Let Z’ = D(g)(1@M(-3)) and Z” = D(g)(1@M(—3)). Then 
Z' and Z" are irreducible D(g)-modules and we have a decomposition [4(M) = 
Z' @Z". As (G,D(g)°)-bimodules 


; 3 
Z' = @— Fx ® M(-k- 3) 
k>0 


and 
1 
Zl" _ iy 
D Fx ® M(k— 5) 
k>0 


Furthermore, Z’ is isomorphic with O(g) as a D(g)-module. 


7. Duality and a decomposition result 


We now introduce another module, M, which will play an important role in 
the study of M. Let M be defined dually to M as 


M = D(g) @s(h-@n_@n,*) C- 


Let X1,...,X» be an orthonormal basis of g (with respect to a G-invariant non- 
degenrate symmetric bilinear form on g). Let 11,...,£, be the corresponding 
linear coordinates. Define a linear anti-automorphism D ++ D? of D(g) by 
ti Xj, X; ++ x;. We assert that there exists a non-degenerate pairing (, ) of M 
with M which is D(g)-contravariant, ie. (Du,v) = (u, D'v) for u€ M,v € M. 
Lett V =h@n* @n,. Then V* = h* O@n_ On¥# and gGg* = V GV* and 
the natural pairing of S(V) with S(V*) induces a pairing of M with M with the 
required properties. 


LEMMA 7.1. The non-degenerate pairing of M with M induces a non-degen- 
erate pairing of T7(M) with T?"-7(M). 
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PROOF. On O(G) we define an inner product (, ) as follows. Let 


O(@) = Hoe 


YEG 


be the decomposition of the G-module O(G) into isotypic components relative 
to the action g (or p,). Let R : O(G) —+ O(G)[0] be the projection onto 
the invariants O(G)° = C. If f;h € O(G) then we set (f,h) = R(fh). This 
defines a G-invariant inner product on O(G). We obtain a product pairing 
(which we also denote by (, )) of M@O(G) with M ®O(G). Next we define a 
non-degenerate pairing of A?(g/h) with A2"-3(g/h). Let wo be a fixed non-zero 
element of A?"(g/h). If a € A?(g/h), 8 € A?”-I(g/h) then we define (a, 3) by 
aN B= (a, B)wo. Let &,... ,€¢ be a basis of AJ(g/h) and let €1,... ,€4 be the 
corresponding dual basis of A?"~J(g/h). Now, if w €T7(M),n © T?"-3(M) then 
we set 


w, n) = Dw); n(€)). 


One checks that this defines a non-degenerate D(g)-contravariant pairing of 
TI(M) with T?2"-7(M). O 

We now consider the module N = I'’(M) and its dual N = I'"(M). The 
invariants N© as a D(g)°-module are isomorphic with O(h) with D(g)° acting 
on O(h) via 6. (This follows from the observation that the trivial infinitesimal 
character occurs in M only in the direct summand M(—2p) ® O(h).) Similarly, 
the invariants N©@ are isomorphic with S (h). The pairing of N with N induces 
a pairing of N° with N@. 

In [Wal] we studied the module O(h) as a (D(h)”, W)-bimodule. We record 
the main result. Let W denote the set of equivalence classes of irreducible finite- 
dimensional representations of W. If y € W we fix V, €. To each y € W there 
corresponds a D(h)”-module V7 such that 

i) If V7 is isomorphic with VY as a D(h)”-module then 7 = 7’. 
ii) As a (D(g)”, W)-bimodule O(h) is isomorphic with 


QB v’eV,. 


yew 


LEMMA 7.2. Let V be an irreducible D(g)°-module. Then, up to equivalence, 
there exists exactly one irreducible (D(g),G)-module L(V) such that L(V)° is 
isomorphic with V as a D(g)°-module. 


PROOF. We define a (D(g), G)-module, M(V), as follows: 
M(V) = D(g) Qpig)¢ V 


with G acting on the first factor. Then M(V)° =1@V. Clearly, if X is any 
irreducible (D(g), G)-module such that X© is isomorphic with V as a D(g)°- 
module then X is isomorphic with a quotient of M(V). We assert that M(V) 
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contains a unique proper maximal (D(g), G)-submodule. Set 
Z={ZCM(V)|Z is a (D(g), G)-submodule such that Z° = 0}. 
If Z;,Z. € Z then Z; + Zs eZ. Hence 
M(V)=S°2Z 


ZEz 
is a proper (D(g), G)-submodule of M(V). We note that if X is a (D(g), G)- 
submodule of M(V) such that X° #0 then X° 5 1@V, so X = M(V). Hence 
M'(V) is the unique proper maximal (D(g), G)-submodule of M(V). Now, up 
to equivalence, 
L(V) = M(V)/M"(V) 

is the only irreducible (D(g), G)-module such that L(V)° is isomorphic with V 
as a D(g)°-module. O 


REMARK. If X is a compatible (D(g),G)-module with G connected then X 
is irreducible as a (D(g), G)-module iff it is as a D(g)-module. 


We define a compatible (D(g), G)-module as follows. Let N’ = D(g)NO/R 
with R = {n € D(g)N® | (n, D(g)N°) = 0}. Note that N’ = N if D(g)N° = N 


and D(g)N¢ = N. 
THEOREM 7.3. The compatible (D(g),G)-module N' is isomorphic with 


— (dim y)N7 


yew 


where the N7 are irreducible pairwise non-isomorphic D(g)-modules. Further- 
more, (N7)° is isomorphic with V7 as a D(g)°-module. 


Proor. Let N° = V,@---@V,, be a decompostion of the D(g)°-module NG 
into irreducible submodules. We set N; = D(g)V;. Clearly, N° = D(g)°V; = Vi. 
Let Nj denote the image of N; in N’. Since RM N° = 0 we still have that 
(Nj)© is isomorphic with V; as a D(g)°-module. Let X Cc N/ be a proper 
D(g)-submodule. Then X© is isomorphic with a proper D(g)°-submodule of Vj. 
Hence X° = 0. This implies X | N@, so X 1 D(g)N©. Hence X = 0. Thus, 
the modules Nj are irreducible and the sum 5°; Nj = N’ is direct. The result 
now follows from the observations above. OU 


8. Conjectures 


We will first study the consequences of the following 


CONJECTURE 8.1. 


i) The module M is n_-free. 
ii) The module M is n,-free. 
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REMARK. We proved this conjecture for the simple Lie algebras of type 
A;,A2,A3 and Bz = C2. (The proof for Bz = C2 is a computer proof.) 
However, our technique breaks down for Go. 


THEOREM 8.2. Assume that Conjecture 8.1 is true. Then T?(M) = 0 for 
Zr. 

ProoFr. One can show that if M is free as an n_-module then M admits 
a Verma composition series, i.e. there exists a sequence of g-submodules 0 = 
M_; C Mo C M, C -:- such that M = UL, M; and each M;/M;_1 is a Verma 
module. This implies [7(M) =0 for j < r (cf. [RRG, Lemma 6.4.2]). Similarly, 
if M is free as an n,-module then M admits a Verma composition series (relative 
to the opposite Borel subalgebra). Hence I¥(M) = 0 for j < r. So T#(M) =0 
for 7 >r by Lemma 7.1. O 

We now study the non-zero module N = T'"(M) as a D(g)-module. First we 
recall some notions. Let V be a vector space over C of dimension n < oo. The 
Bernstein filtration of D(V) is defined by 


DV). = S> S*(V*)S9(V). 


itj<k 





With this filtration D(V) is a filtered algebra with corresponding graded algebra, 
gr D(V), isomorphic with O(V x V*). Let M be a D(V)-module. A filtration 
Mo Cc M, C -:: of M is called a D(V)-filtration if D(V);M; C Mi+;. To 
each filtered D(V)-module M there is an associated graded module gr M over 
grD(V). If M admits a D(V)-filtration such that gr M is a finitely generated 
gr D(V)-module (a good filtration) and such that 


C 
dim M;, = ae + terms of lower order 


then M is called holonomic. The following result is due to Bernstein (see [Eh, 
Prop. 1.12 and 1.15] for a proof). 


THEOREM 8.3. Suppose that M admits a (not necessarily good) D(V )-filtra- 
tion such that 


dim M, < mi for k > 0. 


Ifp <n then M = 0 and if p=n then M is holonomic and the length of the 
module M is bounded by C. 


We return to the case V = g of the previous sections. Let M and M be 
the modules defined in Sections 4 and 7. Our goal is to prove that the module 
N = T1'(M) is holonomic (under the assumption that Conjecture 8.1 holds). 
We fix some notation. Let a),...,a, be the postive roots and assume that 
Q,...,Q@, are the simple roots. Set Q = ys Za; and Qy = Se Na;. For 
A= ae nia; € Q then the number ht \ = ya; n; is called the height of A. If 
K € N” then we define (K) = S77_, kia. We note that |K| = )o7_, ki < ht(K). 
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LEMMA 8.4. dim M?!,,_, <C((|é|+ ht A)??7?. 
PRooF. From Lemma 4.1 we obtain 
dim M%y,_y 
= #{(1, J, K) € N' x N" x N"| (J) 4+ (K) =A, i=[J|+|J| —|K]} 


|d|+ht 
= SO Aen’ | |J| =p} 
p=0 

x #{(J, K) EN” x N"| (J) + (K) =A, t= pt |J| - |K]} 
Clearly, #{7 € N! | [J| = p} < (p+ 1)'-?. Similarly, for each p we obtain the 
estimate 

#{(J,K) EN” x N"| (J) + (K) =A, i= pt |J|—|K]} 
= >> HU, K)EN XN |J+K=L, i=pt|J\-|K}} 
(L)=\ 


< S$ (btA+1)" 


(L)=A 
< (bt A+ 1)°'(ht A+ 1)" = (ht A +1)?" 


Thus, 
dim Mi, < ((lé] + ht A) + 1)'(ht +. 1)?7-'-1 < O(a] + ht d)?771. 
This completes the proof. O 


THEOREM 8.5. Assume that Conjecture 8.1 is true. Then N is a holonomic 
D(g)-module. 


Proor. If A € Q is dominant then we fix a finite dimensional irreducible 
(g,G)-module F', with highest weight A. This defines a bijection of the set of 
dominant weights in @ with the set of equivalence classes of irreducible finite- 
dimensional (g, G)-modules. Let N*{A] denote the A-isotypic component of the 
(g, G)-module N* = '7(M*). We note that 


(geg)NiIA]C So (NA+ a] +N [A+a]). 
ae buU{0} 
Let 3 be the highest root. Then we define 
M= >, N'A. 
|i],ht A<k ht B 


The sequence No C N, C --- is a D(g)-filtration of N. We investigate the 
dimensions of the spaces N,. First we observe that 


dimN,=  S>  dimF,-dimHome(Fa, N"). 
ji].ht A<k ht B 
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From Weyl’s formulas we derive the estimate dim(F,) < Cy(ht A)”. Thus, 


dimN, <Cik”- S>  dimHome(Fa, N’). 
ji],ht A<k ht B 


We now consider the characters of the modules M*. Under our assumption we 
may write 
chM* = © ai, ch M(—2p— ) 
AEQy 
with ai, € N. If A € Q is dominant then a,;,, gives an upper bound for 
dim Homcg (Fr, N°). Set aig = om \=j Qi,d- Then 


kht@ 
i < ui ij). 
dim N;, < C,k" (2k ht B+ 1) BOR of ai;) 


Before we can complete the proof we need some preparation. We filter the 
modules M* as follows. Set Mj = Dney<pntg Miop—,- Similarly, for each 
Verma module M(A) set M(A)p = Dont u<pntg M@OA)a-n- 


CLAIM. dim M(A)p > Cop’. 


PROOF OF THE CLAIM. Set M(A)(o) = M(A), and for p > 0 inductively 
M(A)(p) = M(A)p—1) + GM(A)(p-1). Clearly, M(A)(p) C M(A)p. One checks 
that Fi 

dim M(A) (p) = Pr + terms of lower order > Cop’. 


The claim follows. 
Now, 


dim M3 = > aj,,dim M(—2p — d)p-ner 


AEQy 
P 
> C2 So aia(p— ht r)” = C2 >> aij(p— 5)’. 
AEQy j=0 
Replacing p by 2p we find 
2p P P 
dim M3, > C2 S— aij(2p — 7)” > C2 S- ai;(2p— 3)” > Ca¥ agp”. 
j=0 j=0 j=0 


On the other hand, in light of Lemma 8.4, we have the estimate 


max(dim M3,) < Cap*"***. 
i\<p 
Thus, 


P 
max Ga etigs 
max), ii) a” 
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We finally obtain 
dim Ny, < Ck" (2k ht B + 1)Cy(kht B)’t!! < CkK?rt!, 


Since 2r + 1 = dim g we conclude that N is holonomic by Theorem 8.3. 0 
Next we introduce another conjecture. 
CONJECTURE 8.6. 
i) D(g)NG =N. 
ii) D(g)N° =N. 
Theorem 7.3 implies 


THEOREM 8.7. Assume that Conjecture 8.6 is true. Then the compatible 
(D(g), G)-module N is isomorphic with 


B (dim y)N7 
yew 
where the N” are irreducible pairwise non-isomorphic D(g)-modules. Further- 


more, (N7)° = V7 as a D(g)°-module. 


REMARK. The assertion in Theorem 8.7 is actually equivalent to Conjecture 
8.6. 
We now show that Conjecture 8.6 is true for g = slo. 
PROOF OF THEOREM 6.1. We have seen in Section 6 that 
3 1 
= M(-k—- =) ®@ M(k—- -)). 


Let Z°8" = D(g)(1@M(—3)) and Z! = D(g)(1@M(-4)). The irreducible D(g)- 
module N%®" (see Section 7) is a quotient of Z°8". We note that N*8" = O(g) as 
a D(g)-module. In Section 6 we have also seen that 


3 
Ns" = CD Fy, @ M(-k- 5) 
k>0 
as a (G, D(g)°)-bimodule. This implies 


3 


28" 5 QD Fo ® M(-k - 5) 


k>0 
and (since N*8" is irreducible) 
1 
Zo @ Fax @ M(k- 5): 
k>0 


Thus, we may write 


1 
Z* = & For ® M(k— 5) 
kes 
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with S c {0,1,2,...}. We now observe that 


1 
(g @ g*) For @ M(k — 3) 


involves the G-types Fox42, Fo, and Fo,_2. Thus if ko € S and kg +1¢S then 
the subspace 


1 
“= . Fa, ®@ M(k- = 

SO i 
is D(g)-invariant. If we use the same filtration of N as in the proof of Theorem 
8.5 we find that dimZ; < Cj*. (Here Z; = N; MZ.) Theorem 8.3 would 
imply that Z = 0. Hence if kg € S then ky +1 € S. Since 0 € S we see that 
S ={0,1,2,...}. This implies N = Z°8° + Z! = D(g)N°. The same arguments 
replacing O(g) with S(g) prove the assertion for N. O 


REMARK. Sam Evens has shown that the results in [HK] imply that the 
assertions in Theorem 8.2, Theorem 8.5 and Theorem 8.7 are true for a module 
V whose construction is geometric and analogous to our construction of N. 
Thus, if we can show that V is isomorphic with N then the consequences of our 
conjectures would be true. 
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Spectral Zeta Series of Rank 1 Space Forms 


FLOYD L. WILLIAMS 
Dedicated to Ray A. Kunze 


ABSTRACT. We give an alternative proof of Weyl’s law (Gelfand’s Conjec- 
ture) on the asymptotics of the class 1 spectrum, and we give a factorization 
of Selberg’s zeta function. 


1. Introduction 


Let Xp =I \ X be a compact space form of an irreducible rank 1 symmetric 
space X = G/K of noncompact type where K is a maximal compact subgroup 
of the simple group G and [ is a discrete subgroup of G. Fix a finite-dimensional 
unitary representation x of [ and let y also denote the corresponding charac- 
ter. Then the induced unitary representation 7, = indr+cxy of G decomposes 
discretely: 


(1.1) Ty = ye m,(T)1, 


nEG 


G denoting the unitary dual of G, where each multiplicity m,(I’) > 0 is finite. 
Let {7;}j;>0 C G be the subset of all class 1 representations (with respect to K) 
such that n;(x) def mr, (I) > 0. Attached to the class 1 spectrum {7;,7;(x)};>0 
of 7, is a spectral zeta series of Minakshisundaram-Pleijel type 


co 
73(x) 
1.2 Dr(s; 6, = aI NAL = 
with parameter b > 0 where {A;}j>0 is the spectrum of the Laplacian —Ar on 
smooth sections of the vector bundle over Xp induced by x. The eigenvalues A; 
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depend on y and are described in (2.6) below. Dr(s; b,x) converges absolutely 
for Res > d/2, d = dim X; cf. [10, 14, 19]. We assume Xp is a manifold. 

We find an explicit meromorphic continuation to the whole complex plane of 
s — Dp(s;b,x) in terms of the harmonic analysis of X. The result, Theorem 
3.7, is simpler and more general than that obtained in [21]. Combining Theorem 
3.7 with a Tauberian theorem of Delange we obtain a quick proof of a Weyl 
asymptotic law for the class 1 multiplicities {n;(x)};>0 (see Theorem 3.10) as 
conjectured, in principle, by Gelfand [11] and first proved by Gangolli [7, 8]. 
Eaton [5] and Wallach [19] also have obtained proofs where, more generally, T 
is allowed to have torsion (as in [7]). On the other hand our method of proof 
extends, in principle, to noncompact finite volume quotients Xp quite naturally 
(and to I’ with torsion; see remarks following Theorem 3.15), given the theory of 
Eisenstein series. We do hope to push further in this direction. Theorem 3.7 has 
other applications. It leads, for example, to a general factorization of Selberg’s 
zeta function attached to Xr; cf. Theorem 3.15. 

In 1972, the author wrote of his thesis advisor Ray Kunze: “The debts we 
owe him for encouraging and guiding our career, over a prolonged period of time, 
will extend through future years. So will our deep admiration of him.” Some 20 
plus years later, these words indeed continue to echo truth. With much affection 
we dedicate this paper to Ray. 


2. Some Further Notation 


Let G = KA,N be an Iwasawa decomposition of G and let go, ap,9 denote 
the Lie algebras of G, Ap, N. For @ the Cartan involution of G (with respect to 
K), 9 = 9§ the complexification of go, and a D ap a 6-stable Cartan subalgebra of 
go, let ®* be an apy-compatible system of positive roots in the set ® of nonzero 
roots of (g,a©). Then for the set of restrictions E+ = {alapla € P*} where 
Pt = {a € ®*|a £ 0 on ay} one has (as dim ap = 1) 


(2.1) ro = > ga, B* = {B} or {8,26} 


aeht 


with gq a (real) restricted root space. We normalize a choice of basis vector Ho 
of ap by B(Ho) = 1 and we set 


1 
(2.2) OE 5 S> a, po = p(Ho). 
aePpt 


Let H : G — a, be the smooth map such that each x € G has an Iwasawa 
decomposition « = k(x) exp H(x)n(x) € KA,N. 
With the above notation in place we now specify the normalization of Haar 
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measures used henceforth: For compactly supported continuous functions f,h, w 


I. fla yaa = fi f(exptHo)dt, fe —20(H(62)) dn = 1, 


(2.3) [x h(x)dx = I [. [im h(kan)e?('°8 ®) dkdadn, 
i, voide = 5 ay u(t Bat, 


where aj, is the dual space of ap, dt is Lebesgue measure on R, and dk is normal- 
ized Haar measure on K. As G,T are unimodular there is, given dz, a unique 
G-invariant measure mp on T'\G such that for h € C.(G) 


(2.4) / h(x)dx = | S— h(yx)| dmp (Pz). 
G I\G |ser 
Write vol([ \ G) = Ine 1dmry as usual. 
Each of the class 1 representations 7; of section 1 corresponds to a positive 
definite spherical function ¢; which has the form ¢; = ¢,, for some v; € an 
where for v € Oe a € G, by [1], 


(2.5) $.(c) = i olive H(tk)) df, 
K 


Since 7; depends on x, v; does also and we may write v; = v;(y). We take 
m™ = the trivial representation: y% = ip. The 7; with v;(Ho) € iR — {0} 
are finite in number and belong to the spherical complementary series. The 
eigenvalues A; in (1.2) of —Ap are given by 


(2.6) dy = Ag (x) = 95 + ¥5(x)?(Ho). 


Therefore Ag = 0 and we take 0 = Ap < Ay < An <<... 3 limjoo Aj = 0. 

Finally we introduce some notation used in the trace formula. As TI is co- 
compact and torsion free one can find for each y € P— {1} t, >O and m, € M, 
the ceutralizer of A, in K, such that 7 is G-conjugate to m, expt,Ho, with my 
unique up to conjugation in M. By [20] in fact 

e'y = max{|c|:c = an eigenvalue of Ad(7): g > g} 
(2.7) if St = {@} in (2.1), with |e| replaced by |c|? if 2+ = {8,26}; 


see Table 1 below. 
The following function C on I — {1} is therefore well-defined: 
(2.8) C(y)7! = etrPo |detn, (Ad(m, exp tHy)—! x 1)| 


Also each y € Tr — {1} has a unique representation y = 69(”) where 6 €T — {1} 
is a primitive element and j(7) > 1 is an integer; i.e. 6 cannot be expressed as 
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aad with 7, €T,7 > 1 an integer. Let Cp be a complete set of representatives in 
I of its conjugacy classes. 


3. Meromorphic Continuation of Zeta 


Following the method of Randol [15], who considered the case G = SL(2, R) = 
SO, (2,1) with x = 1, we can find for any split rank 1 simple Lie group G as above 
an explicit meromorphic continuation to C of s + Dr(s;b, x), using Selberg’s 
trace formula and explicit knowledge of the Plancherel measure |c(r)|~?dr of X 
where c(-) is Harish-Chandra’s c-function. 


Up to local isomorphism we take G = SO (2n,1) or SO,(2n + 1,1) with 
n > 1, SU(n,1) or SP(n,1) with n > 2, or Fyy_20). Given z € C fixed define 
the following polynomials II(r; z) in r of degree (d — 2)/2 for G 4 SO\(2n+1, 1) 
and of degree (d — 1)/2 = n for G = SO,(2n + 151): 


2 
r+ (n-5+35) | for G = SO,(2n,1) (n > 1) 





| for G = SU(n,1) (n > 2) 


2 
pe wages =e 
4 Iv 9 4 























(3.1) areaG By. i for G = SP(n,1) (n> 2) 
moo [ie() -_] fie @)-i] [iQ] 
i+G) We@ {| = Fam 
=[Ifr+o-9) — 2] for G = SO,(2n + 1,1) (n> 1). 


By definition, II(r? + z;z) is an even polynomial P(r) in r, independent of z, 
of degree d— 2 for G 4 SO,\(2n + 1,1), and of degree d— 1 = 2n for G = 
SO,(2n + 1,1). With our normalization of measures, Miatello’s computation 
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[13] of Harish-Chandra’s Plancherel density takes the form 
CerrP(r) tanh ar for G = SO,(2n, 1) 
tanh 5r 


for G = SU(n, 1), with the 
CenrPi(r) or 


(3.2) |e(r)|-? = cotangent choice for n even 
coth 5r 
CetrP(r) tanh Fr for G = SP(n,1) or F420) 
ConP(r) for G = SO\(2n + 1,1) 
where 


Co = Rina, = Tiny (2 Pony), 
(3.3) Q™r(9)?)-1, 2r(at 5) 


in the respective cases. 
Our interest is in the case z = b + p? with b € C fixed. We define coefficients 


my;(b) by 
(3.4) I(r; 6+ po) = Smior 


where m = (d — 2)/2 for G # SO,(2n + 1,1), m = (d-1)/2 = n for G = 
SO,(2n + 1,1). po is given in Table 1. 


Table 1 





G (up to local isomorphism) d 








t- 


SU(n,1),n>2 2n = {8,28} F wer 
SP(n,1),n>2 4n {8,26} & aos 
SO,(2n,1),n>1 5 mn {B} 1 1 
SO,(2n+1,1),n>1 Q2n+1 {BG} mn(b} = 1 
F420) 16 {8,28} § ~ 











One more definition is needed: 
1 v4 2 
(3.5) 6r (t; b) — » x(y)tyj (y) 2C (yen (Pe rebE+ES /4#) 
yeCr—{1} 


for t,b > 0; see definitions (2.7), (2.8). The behavior of Op(t;b) at 0,00 is given 
by 
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LEMMA 3.6. J constants c,a,C(b) > 0 3 limy_.o+ Or(t; b)e*/* = 0, |Op(t;b) — 
no(x)e7 "| < C(b)e~%* for t > 1. 


The first statement requires the asymptotic result lim,» Bre-97 E(x) = 1 
for suitable 8 > 0 where E(x) = |{y € Cr — {1}|t, < x}|; see [2, 8]. One can 
in fact take 8 = 29. Lemma 3.6 is proved in (21] in the case y = 1. The same 
proof works for general y. 

Here’s the main theorem, where we exclude the cotangent case in (3.2) for 
simplicity of statement. For the latter case see remarks following Theorem 3.15. 


THEOREM 3.7. For b > 0 let Tr(s;b,x) = Jo” Or(t;b)t8"1dt, which is an 
entire function of s by Lemma 3.6. For a > 0 let K(s;b,a) = f,(b+ po + 
x*)~’sech? ax dx, which is also an entire function of s. For s € C define 


Tr(s; }, oats “) 








ér(s;b, x) = T(s) Cex(1)volT \ G)- 
“7 K(s—j —158,a(G)) 
» m;(b) ea : 
(3.8) 
$6) (s:b, x) = TDN 4 cay(1)nol \ G)0-+ 02) “sth, 


Ts) 


ils Fe j =~ 5) 
m,;(b)(b + n?)? —__~— 2". 
. il T'(s— J) 
see (3.1), (3.2), (3.3), (3.4), and Table 1. Then for Re s > d/2, Dr(s;b,x) = 
ér(s;6,x) for G = SO,(2n,1), SP(n,1), Fs—20), or SU(l,1) with | odd (up to 
local isomorphism). For G = SO,(2n+1,1) we have Dr(s;b, x) = bh) (5; b, x) 
for Res > d/2=n+4+1/2. 


gr( 3,x), gh) ;b,x) provide therefore the meromorphic continuation of 
Dr(;6,x). We see, for example, that ¢r( ; b,x) is holomorphic except for pos- 
sibly a simple pole at s = p,p = 1,2,... ,d/2, with residue 4Cgx(1)vol(T \ 
G)my_1(b) (since K(0; b,a) = 2/a). In articular ™Ma/2)-1(b) # 0 hee Table 1) 
= d/2 is indeed a simple pole. Also since 1/I'(s) vanishes at s = —1, —2,—3,. 
we can easily compute the value of ¢r( ; 6, x) or of gh ( 3b, x) ai any neaakiee 
integer. Justification can be given for the use of Fubini’s theorem which leads 
to an expression of Tp in terms of the K-Bessel function K_: 


(3.9) Tr(s; 6, x) 


=X xenstn teens ay (tn/o+ 8) 


Sry 
lave 08) ° vecr—0} 


As an application of Theorem 3.7 we derive 
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THEOREM 3.10. Let w denote the Weyl function 


(3.11) xr w(zr yd Son; ) for x«>0. 
Aj Sz 


Then w is finite-valued and w(x) ~ (Cg/4)x(1)v0l(T \ G)myaj2)-1 (1) @2/?_ as 
x — 00, with m(aj2)-1(1) replaced by mrn(1) =1 in case G = SO\(2n+1, 1); see 
(3.3), (3.4), and Table 1. 


PRooF. Observe first that, after all, Dp(s; b, y) is a Dirichlet series 


CO 
So nj(x)e* 
j=0 


for 3; def log(b + A;). pines peo 1 (x)e~*? converges, say for ¢ = (d/2) + 1, 
that is limj;_..n;(x)e7%” = 0, given x > 0 J an integer N, > 0 ‘ j2Nz > 
nj(x)e7 7 < ellesth+=)lo_ Since each n;(x) > 1 we can have ; < x only for 
j < Nz. w(x) is therefore finite (as Gangolli has already sbeswedy: We choose 
b = 1 (ie. $89 = 0) and represent our Dirichlet series as a Laplace-Stieltjes 
transform: 


[ e **da(t) = —no(x) + So nj (x)e 
j=0 


(3.12) 


for a(t) def Ss; n;(x) = w(e! — 1), t > 0. 
By St 


That is, by Theorem 3.7, dar e *tda(t) has the form 





(3.13) 7 e *‘da(t) = aa oS +h(s) forRes>a 
0 

where a = d/2, g and h are holomorphic for Res > a, p # 0, —1, —2,-3,..., 
and g(a) # 0. Namely, p = 1 and for Ar(x) = (a(G)/8)Cex(1)vol(T \ G), 
g(s) ® Ar(x)myay2)—1(1)K(s—4/2; 1, a(G)) for G # SO, (2n+1, 1) and g(s) (i) 
Br(1+n?)-8t™41/2T(s —n + 1/2)/T'(s —n) for G = SO,(2n + 1,1) (using that 
I(s—n-—1/2)(s —(n+1/2)) =T'(s —n+1/2)), where Bp(x) = (a1/2/4)Cox(1) 
-vol([ \ G). Now given the format (3.13) one concludes by Delange [3], for 
example, that a(t) ~ (g(a)e%t?—1)/T(p) = g(d/2)e(4/" as t + 00. By (i), 
(ii), (4/2) = (1/4)Cex(1)vol(P \ G@)m(a2)-1(1) with ma72)-1(1) replaced by 
Mn(1) = 1 in case G = SO;(2n+1, 1) (again using K(0;1,a) = 2/a). By (3.12), 
w(x) = a(log(a + 1)) ~ g(d/2)(a+ 1)4/2, or w(x) ~ g(d/2)x4/? as desired, since 
(2+ 1)¢/? ~ 24/2, O 

As another application of Theorem 3.7 we give a factorization of the Selberg 
zeta function Zp(-, x) attached to X7y; cf. [9, 17, 25]. Let Ap be the Laplacian of 
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section 1 and again let Ar(x) (iti) (a(G)/8)Cex(1)vol(T \ G). For a,b > 0,c € 
C,n=0,1,2,3,..., define 


(3.14) I,(b, a; ¢) = i t?"(sech? at)(b? + t*) [log(6* + t?) — ¢] dt. 
R 


The integrals J,, are evaluated in [24], where it is shown, in particular, that 6 - 
[,,(6, a; c) admits a holomorphic continuation to the domain C—{r+iy|y = 0,2 < 
—a/2n}. The following result which we state for Re s > 29 therefore extends to 
a larger domain. For simplicity we omit the cases G = SO\(2n + 1,1), SU(I, 1) 
with | even. 


THEOREM 3.15. 3 c(x) > 0 such that 


(3.16) Zp(s,x) = e(x) det [Ar + 5(s — 2po)l] [e™ale'e-2A0))Fo(9-p0.20G)) 


Ar(x) 


Nia 


™m 


TT! (Pee (2) (s—p0)?4-Y (spo ,a(G)s(1/(5+1)))) 


see (3.4), definition (iit) preceding (3.14), and (3.14). 


Here det(—Ap + 61) def e~(2/9s)¢r(0:b.x) (which is well-defined by Theorem 
3.7; also see (14]) is the usual regularization of the determinant. Theorem 3.15 
was obtained for G = SL(2,R) by Sarnak [16], and Voros [18]; cf. [4, 6, 12] 
and other references in {23], where Theorem 3.15 is proved (in case x = 1). By 
working with the coefficients m,;(6) in (3.4) rather than the coefficients of P(r) 
(which we called Miatello coefficients in {22, 23]) we obtain simpler formulas in 
Theorems 3.7, 3.15. 

For G locally isomorphic to SU(I,1) with I even (the cotangent case in (3.2)) 
we modify the definition of ér(s; 6, x) in (3.8) by adding to it the following extra 
term: (1/4)x(1)vol(T \ G)Ce \25?/?" my (b)Lo(s — j; 6, a(G) = 1/2), where 
fora >0 


(3.17) 





Ghar if (x esch ax) sech ax dx 
Rn (b+ p§+2°)* 


Then Theorem 3.7 holds. Here the integral in (3.17) converges uniformly on 
compact subsets of the plane (as one has an estimate xcesch ax < Me~@!*!/2) and 
is thus an entire function of s. The format of equation (3.13) is thus unchanged 
and the proof of Theorem 3.10 is thus unchanged in the cotangent case. 

In case IT’ is allowed to have torsion there are central and elliptic contribu- 
tions to the trace formula (in addition to the usual hyperbolic contribution). 
For the elliptic contributions (finite in number) we must show that the Mellin 
transform s > [5° ,(h)t*~‘dt of the orbital integral [,(h;) of the fundamen- 
tal solution h; (t > 0) of the heat equation on X is holomorphic, for elliptic 
y € IT. Then as the central contributions are trivially handled, one obtains 
w(x) ~ | center of G|(Cg/4)x(1)vol(T \ G)ma/2)-1(1)24/? in Theorem 3.10. 
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Throughout, our symmetric space is assumed to have rank equal 1, as in [5, 
8]. In [7] G is assumed to possess a complex structure, and in [19] the rank is 
arbitrary. In all cases [ \ G is assumed to be compact. 

Added in proof. In the four months since this paper was written the author 
has essentially worked out the extension of the main result Theorem 3.7 (and 
hence Theorem 3.10) to cover the case of [ \ G with finite volume. 
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